Image processing device

ABSTRACT

Provided is an image processing device for sorting frames included in one moving image into different channels, the frames included in the moving image being a mixture of frames belonging to the different channels. The image processing device includes an automatic sorting unit. The automatic sorting unit calculates a similarity degree of the plurality of frames included in the moving image by performing image processing on the frames, and sorts the plurality of frames into the plurality of channels based on the similarity degree.

TECHNICAL FIELD

The present invention relates to an image processing device and an imageprocessing program, and particularly relates to an image processingdevice and image processing program that are configured to sort framesincluded in one moving image into different channels, the framesincluded in the moving image being a mixture of frames that belong tothe different channels.

BACKGROUND ART

An intermittent recording method, called “time-lapse”, is known as amoving image recording method. This is a method for recording an imageat a lower frame rate than usual, that is, in a time-lapse manner, andenables recording for a longer time period than usual to the samerecording medium. Security cameras often employ the time-lapse method,and furthermore a video picture from another security camera may beinserted into a time period gap that is generated by lowering the framerate. In this case, a moving image in which video pictures from aplurality of cameras, that is, video pictures of a plurality of channelsare switched alternately along one time-line, is recorded.

With respect to the moving image in which video pictures of a pluralityof channels are recorded together on one time-line, video pictures fromeach camera, that is, each channel, can be displayed on a monitor withdedicated equipment. In the dedicated equipment of Patent Literature 1,channel number data is added to the frames of video pictures of aplurality of channels, and at the time of reproduction, the dedicatedequipment can select only the video pictures of a desired channel fromamong the video pictures of the plurality of channels by referencing thechannel number data.

CITATION LIST Patent literature

Patent Literature 1: IP 2002-319210A

SUMMARY OF INVENTION Technical Problem

However, when reproducing video pictures in which video pictures ofmultiple channels are mixed with each other, using general-purposereproduction equipment, it is impossible to view video pictures for eachchannel. The specifications for recognizing the channel of each framediffers between manufacturers or device types, and thus general-purposereproduction equipment cannot support these specifications.

It is an object of the present invention to provide an image processingdevice and an image processing program that enable, when one movingimage includes a mixture of frames that belong to different channels,reproduction for each channel regardless of specifications by which themoving image is recorded.

Solution to Problem

According to a first aspect of the present invention, an imageprocessing device is configured to sort frames included in one movingimage into different channels, the frames included in the moving imagebeing a mixture of frames belonging to the different channels andincludes an automatic sorting unit. The automatic sorting unitcalculates a similarity degree of the plurality of frames included inthe moving image by performing image processing on the frames, and sortsthe plurality of frames into the plurality of channels based on thesimilarity degree.

Here is provided a function for sorting, when one moving image includesa mixture of frames belonging to different channels, these frames by thechannels. Specifically, first a degree of similarity (similarity degree)between the plurality of frames included in the moving image iscalculated by performing image processing on the frames, and theplurality of frames are automatically sorted into the plurality ofchannels based on the similarity degrees. According to this function,regardless of the specifications of the moving image, it is possible toseparate the moving image that includes a mixture of frames belonging todifferent channels into moving images for fee respective channels. That,is, when one moving image- includes a mixture of frames belonging todifferent channels, it is possible to perform reproduction for eachchannel, regardless of under which specifications the moving image isrecorded.

According to a second aspect of the present invention, the imageprocessing device according to the first aspect is such that theautomatic sorting unit includes a setting unit, a calculation unit, anda determination unit. The setting unit sets a specific frame included inthe moving image or a frame that is obtained by combining two or morespecific frames included in the moving image as a reference frame, andsets another specific frame included in the moving image as acomparative frame. The calculation unit executes calculation processing.The calculation processing is processing of dividing the comparativeframe and the reference frame into a plurality of sub-regions,calculating a local similarity degree between the comparative frame andthe reference frame for each sub-region, and calculating a totalsimilarity degree only using sub-regions having a high local similaritydegree. The total similarity degree is a frame-overall similarity degreebetween the comparative frame and the reference frame. The determinationunit executes determination processing of determining, based on thetotal similarity degree, whether or not the comparative frame belongs tothe same channel as that of the reference frame or the frames combinedinto the reference frame.

Note that “dividing a frame into a plurality of sub-regions” includes,in addition to thoroughly dividing an entire frame into a plurality ofsub-regions without the sub-regions overlapping each other, thoroughlydividing a partial region of an entire frame (for example, the centralregion excluding an outer region, or the like) into a plurality ofsub-regions without the sub-regions overlapping each other, dividing theentirety or a part of a frame into a plurality of sub-regions with somesub-regions overlapping each other, and the like.

Here, in order to calculate the degree of similarity between the framesincluded in the moving image, the comparative frame and the referenceframe are divided into sub-regions, and local degrees of similaritybetween the comparative frame and the reference frame are calculated ona sub-region basis. Then, a total similarity degree for the entire frameis calculated only using sub-regions having a high local similaritydegree. This total similarity degree is used as a criteria fordetermining whether or not the comparative frame and the reference framebelong to the same channel.

That is, here, information relating to the sub-regions having a lowlocal similarity degree is not taken into consideration for thecalculation of the total similarity degree. This is because framesbelonging to the same channel have the same background image except fora part in which a moving subject is present (hereinafter, referred to as“moving subject part”). Also, if the moving subject part accounts for alarge ratio in the entire frame, and the total similarity degree betweenframes is determined using information relating to the entire frame,there is a risk that even the frames belonging to the same channel aredetermined as having a low total similarity degree. Accordingly, here,the sub-regions that are considered to include a large moving subjectpart and have a low local similarity degree are not used in thecalculation of the total similarity degree, in order to preventerroneous determination. As a result the influence of a moving subjectis reduced, and it is possible to correctly determine whether or notcompared frames belong to the same channel.

According to a third aspect of the present invention, the imageprocessing device according to the first aspect is such that theautomatic sorting unit includes a setting unit, a calculation unit, anda determination unit. The setting unit sets a specific frame included inthe moving image or a frame that is obtained by combining two or morespecific frames included in the moving image as a reference frame, andsets another specific frame included in the moving image as acomparative frame. The calculation unit executes calculation processing.The calculation processing is processing of detecting feature pointsfrom the comparative frame, setting a vicinity region for each featurepoint, and calculating a total similarity degree only using the vicinityregions. The total similarity degree is a frame-overall similaritydegree between the comparative frame and the reference frame. Thedetermination unit executes determination processing of determining,based on the total similarity degree, whether or not the comparativeframe belongs to the same channel as that of the reference frame or theframes combined into the reference frame.

Here, in order to calculate the degree of similarity between the framesincluded in the moving image, feature points are detected from thecomparative frame, and the total similarity degree between thecomparative frame and the reference frame is calculated only usingvicinity regions of the feature points. This total similarity degree isused as a criteria for determining whether or not the comparative frameand the reference frame belong to the same channel.

That is, here, information relating to the regions other than thevicinity regions of the feature points is not taken into considerationfor the calculation of the total similarity degree. This is because, ifframes are of different channels, but are of images obtained bycapturing similar scenes, there is a risk that using informationrelating to the entire frame to determine the total similarity degreewill cause a high total similarity degree to be determined. For example,in a case of two video pictures in which different places of the sameshop interior are captured, the major part of the frame of both videopictures may include wallpaper, a floor, and the like in the sameuniform color, and in such a case, the total similarity degree of theentire frame is high. Accordingly, here, in order to prevent erroneousdetermination, the regions other than the vicinity of the featurepoints, that is, the regions in which a background such as wallpaper anda floor in the same uniform color is possibly captured are not used inthe calculation of the total similarity degree. As a result, it ispossible to reduce the influence of a similar background, and tocorrectly determine whether or not compared frames belong to the samechannel.

According to a fourth aspect of the present invention, the imageprocessing device according to the third aspect is such that thecalculation unit divides the comparative frame into a plurality ofsub-regions, and detects the feature points for each sub-region from thecomparative frame.

Here, in order to calculate the degree of similarity between the framesincluded in the moving image, the comparative frame is divided intosub-regions, and feature points are detected on a sub-region basis. As aresult, the feature points are detected substantially uniformly from theentire frame without bias to a partial region of the entire screen ofthe frame. As a result, when the total similarity degree is calculatedonly using the vicinity regions of the feature points, it is possible toprevent a situation in which the total similarity degree is a valueobtained by evaluating the similarity degree of a biased partial region.Accordingly, it is possible to correctly evaluate the similarity degreeof the compared frames.

According to a fifth aspect of the present invention, the imageprocessing device according to the third or fourth aspect is such thatthe calculation unit calculates, for each vicinity region, a localsimilarity degree between the comparative frame and the reference frame,and calculates the total similarity degree only using vicinity regionshaving a high local similarity degree.

Here, in order to calculate the degree of similarity between the framesincluded in the moving image, the comparative frame is divided intosub-regions, and feature points are detected on a sub-region basis.Then, a local similarity degree between the comparative frame and thereference frame is calculated on a basis of a vicinity region of eachfeature point, and furthermore, a total similarity degree is calculatedonly using the vicinity regions having a high local similarity degree.

That is, here, information relating to the vicinity regions of thefeature points having a low local similarity degree is not taken intoconsideration for the calculation of the total similarity degree. Thisis because frames belonging to the same channel have the same backgroundimage except for a moving subject part. Also, there is a risk that, ifthe moving subject part accounts for a large ratio in the entire frame,and the total similarity degree is determined using the informationrelating to the entire frame, even frames belonging to the same channelwill be determined to have a low total similarity degree. Accordingly,here, the regions that are considered to include a large moving subjectpart and have a low local similarity degree are not used in thecalculation of the total similarity degree, in order to preventerroneous determination. As a result, the influence of a moving subjectis reduced, and it is possible to correctly determine whether or notcompared frames belong to the same channel.

According to a sixth aspect of the present invention, the imageprocessing device according to any one of the second to fifth aspects issuch that the setting unit accepts, from a user, setting of an area in aframe screen, sets an image in the area of the specific frame includedin the moving image or a frame that is obtained by combining images inthe area of the two or more specific frames included in the movingimage, as the reference frame, and sets an image in the area of theother specific frame included in the moving image as the comparativeframe.

For example, if the target moving image to be processed is a videopicture from a security camera or the like, it is often the case thatthe frame screen has a date and time display area. If information of theentire frame including such a date and time display area is used todetermine the similarity degree between frames, there may be a risk thateven frames belonging to the same channel are determined to have a lowsimilarity degree. Furthermore, black framing may be provided in theframe screen. In such a case, conversely, if information of the entireframe including the black framing is used to determine the similaritydegree of frames, there may be a list that even frames belonging todifferent channels are determined to have a high similarity degree.Here, in the frame screen, only an area designated by the user is usedas a basis of the calculation of the similarity degree. Accordingly, itis possible to exclude the area that may cause erroneous determinationfrom the areas serving as a basis of the similarity degree calculation.

According to a seventh aspect of the present invention, the imageprocessing device according to any one of the second to sixth aspects issuch that the setting unit is capable of setting a plurality ofreference frames. The calculation unit executes, if the plurality ofreference frames are set, the calculation processing on each of thereference frame. The determination unit determines, in the determinationprocessing, that the comparative frame and the reference frame or theframes combined into the reference frame that have the highest totalsimilarity degree belong to the same channel.

Here, it is possible to set a plurality of reference frames. Also, thecomparative frame is compared with each reference frame, and is sortedto the same channel as that of the reference frame that gives thehighest total similarity degree. Note that the reference frame herefunctions as a frame representing a predetermined channel. As a result,it is possible to correctly sort the frames included in the moving imageinto the plurality of channels.

According to an eighth aspect of the present invention, the imageprocessing device according to the seventh aspect is such that thesetting unit sequentially selects frames from the moving image, and setsthe first selected frame as the reference frame and a frame selectedthereafter as the comparative frame. Each time the setting unit newlysets a comparative frame, the calculation unit executes the calculationprocessing and the determination unit executes the determinationprocessing. If it is determined, in the determination processing, thatthe comparative frame does not belong to the same channel as any of thereference frames or any of the frames combined into the referenceframes, the setting unit again sets that comparative frame as thereference frame.

Here, the frames are sequentially selected from the moving image, andthe first selected frame is set as a reference frame representing apredetermined channel. Furthermore, the frame selected, secondly onwardis first set as a comparative frame. Then, the comparative frame iscompared with each previously set reference frame, and if it isdetermined that the comparative frame does not belong to any of thechannels corresponding to the reference frames, the comparative frame isagain set as a reference frame representing a new channel. According tothis method, it is possible to prevent comparison with, for example, theframes included in the moving image in an all-to-all manner, and to sortthe frames included in the moving image into the plurality of channelswhile suppressing the calculation load.

According to a ninth aspect of the present invention, the imageprocessing device according to the seventh or eighth aspect is such thatthe setting unit sequentially selects frames from the moving image, andsets the first selected frame as the reference frame and a frameselected thereafter as the comparative frame. Each time the setting unitnewly sets a comparative frame, the calculation unit executes thecalculation processing and the determination unit executes thedetermination processing. If it is determined, in the determinationprocessing, that the comparative frame belongs to the same channel asany of the reference frames or any of the frames combined into thereference frames, the setting unit combines that comparative frame withthe reference frame.

Here, the frames are sequentially selected from the moving image, andthe first selected frame is set as a reference frame representing apredetermined channel. Furthermore, the frames selected secondly onwardare first set as a comparative frame. Then, the comparative frame iscompared with each previously set reference frame, and if it isdetermined that the comparative frame belongs to any one of the channelscorresponding to the reference frames, the comparative frame is combinedwith the reference frame. According to this method, the reference framerepresenting a predetermined channel is an image correctly representingvarious frames belonging to the channel, and it is possible to correctlysort the frames included in the moving image by the plurality ofchannels.

According to a tenth aspect of the present invention, the imageprocessing device according to the eighth or ninth aspect is such that,if the plurality of reference frames are set, after the final frame hasbeen selected from the moving image and the calculation processing andthe determination processing have ended, the calculation unit calculatesa similarity degree between the plurality of reference frames. Thedetermination unit combines channels that correspond to reference framesthat have a high similarity degree between the plurality of referenceframes into one channel.

Here, after determination of the automatic sorting with respect to eachframe has ended, the similarity degree between the channels isdetermined again. Accordingly, it is possible to reduce the number ofchannels that are ultimately detected by the automatic sorting and toimprove the accuracy in the sorting of the channels.

According to an eleventh aspect of the present invention, the imageprocessing device according to any one of the eighth to tenth aspects issuch that, after the final frame has been selected from the moving imageand the calculation processing and the determination processing haveended, the determination unit deletes the channel to which only apredetermined number or less of frames belong.

Here, after determination of the automatic sorting with respect to eachframe has ended, if there is a channel to which only a predeterminednumber or less of frames belongs, such a channel is deleted.Accordingly, it is possible to prevent a situation in which a channelthat is not regarded as a respectable channel is included in the finalresult of the automatic sorting, improving the accuracy in sorting ofchannels.

According to a twelfth aspect of the present invention, the imageprocessing device according to any one of the first to eleventh aspectsis such that the similarity degree is a correlation coefficient.

Here, it is possible to sort the frames included in the moving imageinto the plurality of channels, based on the correlation coefficientbetween the frames.

According to a thirteenth aspect of the present invention, an imageprocessing program is configured to sort frames included in one movingimage into different channels, the frames included in the moving imagebeing a mixture of frames belonging to the different channels, andcauses a computer to execute a sorting step of calculating a similaritydegree of the plurality of frames included in the moving image byperforming image processing on the frames, and sorting the plurality offrames into the plurality of channels based on the similarity degree.Here, the same effects as those of the first aspect can be achieved.

According to a fourteenth aspect of the present invention, the imageprocessing program according to the thirteenth aspect is such that thesorting step includes the steps of: setting a specific frame included inthe moving image or a frame that is obtained by combining two or morespecific frames included in the moving image as a reference frame;setting another specific frame included in the moving image as acomparative frame; dividing the comparative frame and the referenceframe into a plurality of sub-regions; calculating a local similaritydegree between the comparative frame and the reference frame for eachsub-region; calculating a total similarity degree, which is aframe-overall similarity degree, between the comparative frame and thereference frame only using sub-regions having a high local similaritydegree; and executing determination processing of determining, based onthe total similarity degree, whether or not the comparative framebelongs to the same channel as that of the reference frame or the framescombined into the reference frame. Here, the same effects as those ofthe second aspect can be achieved.

According to a fifteenth aspect of the present invention, the imageprocessing program according to the thirteenth aspect is such that thesorting step includes the steps of: setting a specific frame included inthe moving image or a frame that is obtained by combining two or morespecific frames included in the moving image as a reference frame;setting another specific frame included in the moving image as acomparative frame; detecting feature points from the comparative frame;setting a vicinity region for each feature point; calculating a totalsimilarity degree, which is a frame-overall similarity degree, betweenthe comparative frame and the reference frame only using the vicinityregions; and determining, based on the total similarity degree, whetheror not the comparative frame belongs to the same channel as that of thereference frame or the frames combined into the reference frame. Here,the same effects as those of the third aspect can be achieved.

Advantageous Effects of Invention

According to the present invention, it is possible to separate a movingimage that includes a mixture of frames belonging to different channelsinto moving images for the respective channels, regardless of thespecification of the moving image. That is, when one moving imageincludes a mixture of frames that belong to different channels,reproduction for each channel is possible regardless of specificationsby which the moving image was recorded.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an image processing deviceaccording to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating a basic screen before image data isloaded.

FIG. 3 is a diagram illustrating the basic screen after the image datais loaded.

FIG. 4 is a diagram illustrating a still image group that belongs to onetime-line.

FIG. 5 is a diagram illustrating an area setting window.

FIG. 6 is a diagram illustrating a re-sorting window.

FIG. 7 is another diagram illustrating the re-sorting window.

FIG. 8 is a diagram illustrating a check window.

FIG. 9 is a flowchart illustrating a flow of automatic sortingprocessing according to the first embodiment of the present invention.

FIG. 10 is a schematic diagram illustrating a method to calculate asimilarity degree according to the first embodiment of the presentinvention.

FIG. 11 is a flowchart illustrating a flow of automatic sortingprocessing according to a second embodiment of the present invention.

FIG. 12 is a schematic diagram illustrating a method to calculate asimilarity degree according to the second embodiment of the presentinvention.

FIG. 13 is a flowchart illustrating a flow of automatic sortingprocessing according to a third embodiment of the present invention.

FIG. 14 is a schematic diagram illustrating a method to calculate asimilarity degree according to the third embodiment of the presentinvention.

FIG. 15 is a schematic diagram illustrating another method to calculatea similarity degree according to the third embodiment of the presentinvention.

FIG. 16 is a schematic diagram illustrating yet another method tocalculate a similarity degree according to the third embodiment of thepresent invention.

FIG. 17 is a flowchart illustrating a flow of processing for evaluatinga color trend according to the third embodiment of the presentinvention.

FIG. 18 is a diagram illustrating impression defining information.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an image processing device and an image processing programaccording to some embodiments of the present invention will be describedwith reference to the drawings.

1. First Embodiment <1-1. Summary of Image Processing Device>

An image processing device 1 of FIG. 1 shows an embodiment of the imageprocessing device according to the present invention. The imageprocessing device 1 is a general-purpose personal computer. An imageprocessing program 2, which is an embodiment of an image processingprogram according to the present invention, is provided and installed inthe image processing device 1, for example, from a computer readablestorage medium 60 such as a CD-ROM, DVD-ROM, or a USB memory that storesthe image processing program 2. The image processing program 2 isapplication software for assisting in image processing to be performedon a moving image and a still image. The image processing program 2causes the image processing device 1 to execute steps included in theoperations that will be described later.

The image processing device 1 includes a display 10, an input unit 20, astorage unit 30, and a control unit 40. These units 10 to 40 areconnected to each other via, for example, a bus line or cable 5, and cansuitably perform communication with each other. The display 10 isconstituted by a liquid crystal display or the like, and displays, to auser, a screen or the like that will be described later. The input unit20 is constituted by a mouse, a keyboard, and the like, and accepts auser operation on the image processing device 1. The storage unit 30 isa nonvolatile storage area that is constituted by a hard disk, or thelike. The control unit 40 is constituted by a CPU, a ROM, a RAM, and thelike.

The image processing program 2 is stored in the storage unit 30. Thestorage unit 30 secures a software management region 50. The softwaremanagement region 50 is a region for use by the image processing program2. The software management region 50 secures an original image region51, a processed file region 52, and an impression defining region 53.The functions of the regions 51 to 53 will be described later.

By reading out and executing the image processing program 2 stored inthe storage unit 30, the control unit 40 virtually operates as anautomatic sorting unit 41 and a re-sorting unit 45. Furthermore, theautomatic sorting unit operates also as a setting unit 42, a calculationunit 43, and a determination unit 44. The operations of the units 41 to45 will be described later in detail.

<1-2. Details of Configuration and Operation of Image Processing Device>

Upon detecting that the user has performed a predetermined operation viathe input unit 20, the control unit 40 starts the image processingprogram 2. After the start-up of the image processing program 2, a basicscreen W1 (see FIG. 2) is displayed on the display 10. Note that displayof all elements such as screens, windows, and buttons displayed on thedisplay 10 is controlled by the control unit 40.

<1-2-1. Image Data Loading>

The basic screen W1 accepts, from the user, an instruction to load imagedata to the original image region 51. The image data loaded to theoriginal image region 51 is to be subjected to reproduction processing,image processing, and channel separation processing, which will bedescribed later. The control unit 40 loads image data from a still imagefile or a moving image file to the original image region 51. Note thatin the present specification, “still image file” refers to a data filein a still image format and “moving image file” refers to a data file ina moving image format.

In the case of loading image data from a still image file, the useroperates the input unit 20 so as to designate one still image file orone folder. In the case of the former, the control unit 40 prompts theuser to input the address path of this still image file in the storageunit 30 and the file name thereof. In the case of the latter, thecontrol unit 40 prompts the user to input the address path of thisfolder in the storage unit 30 and the folder name thereof. Then, thecontrol unit 40 saves the designated still image file or all still imagefiles in the designated folder, as a still image file group, in theoriginal image region 51. Note that in the present specification,“group” does not necessarily include a plurality of elements but may beone element.

On the other hand, in the case of loading of image data from a movingimage file, the user operates the input unit 20 so as to input theaddress path of the one moving image file in the storage unit 30 and thefile name thereof. Upon detecting that the user has designated themoving image file, the control unit 40 displays a moving image loadwindow (not shown) on the basic screen W1 in a superimposed manner. Themoving image load window accepts, from the user, the selection of anarbitrary segment from among the entire segment on the time-line of thedesignated moving image file. Upon detecting that the user has selecteda specific segment via the input unit 20, the control unit 40 generatesa still image file group that has a one-to-one correspondence with aframe group included in the segment of the designated moving image file.Then, the control unit 40 saves the still image file group in theoriginal image region 51. Accordingly in the present embodiment imagedata that is to be subjected to reproduction processing and channelseparation processing, which will be described later, is not of a movingimage file but of a still image file.

Note that even if the still image file group loaded to the originalimage region 51 is derived from a still image file instead of a movingimage file, the control unit 40 recognizes that the still image filegroup is sequenced along one time-line. The sequence is automaticallymade based on a file attribute or the like.

<1-2-2. Reproduction Processing>

After the still image file group is loaded to the original image region51, the control unit 40 displays display windows W2 (see FIG. 3) on thebasic screen W1 in a superimposed manner. The number of generateddisplay windows W2 corresponds to the number of time-lines of the stillimage file group loaded to the original image region 51.

First, one still image file (for example, the still image file of thefirst frame in the time-line) that is included in the still image filegroup loaded to the original image region 51 is displayed in a displaywindow W2. Then, as will be described later, the frame that is displayedin the display window W2 is switched upon an operation of the user.

The control unit 40 can reproduce, within a display window W2, the framegroup that belongs to the time-line corresponding to this display windowW2, as a moving image. Here, as shown in FIG. 3, a window selectionpull-down menu T1, a play button T2, a next frame button T3, a previousframe button T4, and a time-line bar T5 are arranged on the basic screenW1.

Even if a plurality of display windows W2 are present, there is oneactive display window W2. The window selection pull-down menu T1accepts, from the user, the selection of which display window W2 is toserve as the active display window W2. Hereinafter, the time-line thatcorresponds to the active display window W2 is referred to as “activetime-line”, and the frame group that belongs to the active time-line isreferred to as “active frame group”. Furthermore, the frame that iscurrently displayed in the active display window W2 is referred to as“active display frame”.

The play button T2 accepts an instruction to reproduce the active framegroup as a moving image from the user. Upon detecting that the user haspressed the play button T2 via the input unit 20, the control unit 40displays, within the active display window W2, the frames included inthe active frame group sequentially along the time-line in aframe-by-frame format. Note that the reproduction starts from the activedisplay frame at a point in time when the play button T2 is pressed.Furthermore, the play button T2 accepts an instruction to stop thereproduction from the user. Upon detecting that the user has pressed theplay button T2 via the input unit 20 during the reproduction, thecontrol unit 40 fixes the display of the active display window W2 to theactive frame at that point in time.

The next frame button T3 and the previous frame button T4 respectivelyaccept, from the user, instructions to switch the active display frameto the next frame and to the previous frame along the active time-line.

The time-line bar T5 is an object schematically showing the activetime-line. The time-line bar T5 is equally divided as per the number offrames included in the active frame groups in the direction in which thebar extends. The divided region from the left of the time-line bar T5corresponds to the frame of the active time-line (n is a naturalnumber).

As shown in FIG. 3, the time-line bar T5 displays a divided region A1that corresponds to a selected frame group and a divided region A2 thatcorresponds to a non-selected frame group in different modes. “Selectedframe group” refers to a frame group that corresponds to a currentlyselected segment of the active time-line. “Non-selected frame group”refers to a frame group that corresponds to a currently non-selectedsegment of the active time-line.

The time-line bar T5 accepts the selection of an arbitrary segment inthe active time-line from the user. In other words, by operating thedivided region on the time-line bar T5 via the input unit 20, the usercan select an arbitrary number of arbitrary frames from among the activeframe group. The control unit 40 recognizes the selected frame group asa target of image processing and channel separation processing, whichwill be described later. Note that, each time the user selects a dividedregion of the time-line bar T5, the active display frame is switched tothe frame that corresponds to the most recently selected divided region.

<1-2-3. Image Processing>

Hereinafter, the image processing that is to be performed on theselected frame group will be described. The control unit 40 can execute,with respect to the selected frame group, a plurality of imageprocessing modules such as noise removal, sharpness,brightness/contrast/chroma adjustment, image resolution, rotation,addition of characters/arrows/mosaics, and image averaging. The imageprocessing modules are incorporated in the image processing program 2.

The user can select arbitrary modules from the image processing modulesin an arbitrary order for an arbitrary number of times, by performing anoperation on the basic screen W1 via the input unit 20. Each time thecontrol unit 40 detects that the user has selected an image processingmodule, the control unit 40 executes the image processing module withrespect to the selected frame group at that point in time.

As the image processing module is sequentially executed with respect toa frame once, twice, trice, . . . , the frame is sequentially processedto a first-order frame, a second-order frame, a third-order frame, . . .. The 0-order frame corresponds to the still image file saved in theoriginal image region 51. The (m+1)-order frame corresponds to the stillimage file obtained by executing the image processing module once withrespect to the still image file corresponding to the m-order frame (m isan integer of 0 or more). The control unit 40 sequentially generates thestill image files that correspond to the first-order frames onward, andsaves these still image files separately in the processed file region52.

FIG. 4 is a schematic diagram illustrating how the still image groupthat belongs to one time-line is managed by the image processing program2. In FIG. 4, the horizontal N axis denotes the order of frames on thetime-line, and the vertical M axis denotes the order of processing. Eachsquare of FIG. 4 that corresponds to the coordinates (n, m) in the N-Mspace denotes a still image Q (n, m). The still image Q (n, m) is them-order still image of the n-th frame on the time-line (n is a naturalnumber and m is an integer of 0 or more).

The control unit 40 manages, for each frame, the value of the currentlyselected coordinate m as a parameter m. Immediately after the stillimage file group is loaded to the original image region 51, thecoordinate m, has the default value 0. Then, the coordinate m_(s) is ofthe frame is incremented by 1 each time the image processing module isexecuted. Furthermore, the user can freely change the coordinate m_(s)of a selected frame group by performing a predetermined operation viathe input unit 20. Note that executing an image processing module withrespect to a frame refers to executing the image processing module withrespect to the m_(s)-order still image of this frame. Accordingly,changing the coordinate m_(s) means changing the target with respect towhich the image processing module is to be executed. Displaying a framerefers to displaying the still image of the coordinate m_(s) of thisframe. Accordingly, changing the coordinate m_(s) also means changingthe target to be displayed in the active display window W2.

<1-3. Channel Separation Processing>

Hereinafter, the channel separation processing that is implemented inthe image processing program 2 will be described. “Channel separationprocessing” refers to processing for separating a moving image thatincludes a mixture of video pictures belonging to different channelsinto moving images for the respective channels. That is, the movingimage to be subjected to the channel separation processing is typicallya moving image (hereinafter, referred to as “mixed time-lapse movingimage”) into which a plurality of moving images recorded using atime-lapse method are mixed on one time-line in a manner such that avideo picture of each moving image is embedded into a time period gap ofthe other moving image. “Mixed time-lapse moving image” is a movingimage in which video pictures of a plurality of channels are switchedalternately along a time-line.

The channel separation processing is executed on a selected frame group.The execution of the channel separation processing needs a plurality offrames. Accordingly, it is presumed that, in a state in which theselected frame group includes only one frame, an operation button or thelike for starting the channel separation processing is disenabled andthe channel separation processing cannot be started. Alternatively, ifthe selected frame group includes only one frame but there are aplurality of frames on an active time-line, all of the frames on theactive time-line may be set to be subjected to the channel separationprocessing. Hereinafter, the frame group to be subjected to the channelseparation processing is referred to as “target frame group”.Furthermore, the moving image that is constituted by a target framegroup is referred to as “target moving image”.

The channel separation processing includes automatic sorting processingfor automatically sorting a plurality of frames included in a targetmoving image by a plurality of channels, and re-sorting processing inwhich a user corrects a sorting result of the automatic sortingprocessing by a manual operation.

The automatic sorting processing is executed by the automatic sortingunit 41. The automatic sorting processing is processing for performingimage processing on a plurality of frames included in a target movingimage so as to automatically sort the frames by a plurality of channels.Specifically, the automatic sorting unit 41 calculates a degree ofsimilarity between frames, and sorts the plurality of frames included inthe target moving image by a plurality of channels based on the degreeof similarity. The degree of similarity between frames is calculated asan index of whether or not the compared frames belong to the samechannel. As a result of the automatic sorting processing, although theautomatic sorting processing will be described later in detail theplurality of channels are detected. Each channel includes a plurality offrames. Note that as long as a mixed time-lapse moving image issubjected to the automatic sorting processing, a plurality of channelsare usually detected, but there may be a case where only one channel isdetected as a result of the automatic sorting processing. Furthermore,in the automatic sorting processing, a sort label “non-sorted” is givento the frame that is determined as not belonging to any channel.Accordingly, as a result of automatic sorting processing, the channelname of the corresponding channel or the label “non-sorted” is given toeach of the frames included in the target moving image.

The re-sorting processing is executed by the re-sorting unit 45. There-sorting processing is processing for individually sorting anarbitrary frame included in the target moving image to an arbitrarychannel in accordance with a user operation. Specifically, in there-sorting processing, the non-sorted frame that has not been sorted toany channel by the automatic sorting processing can be individuallysorted to a specific channel. Furthermore, the frame that has beensorted to a wrong channel by the automatic sorting processing can alsobe sorted to another correct channel. Accordingly when the userconsiders the result of the automatic sorting processing as wrong, he orshe can correct the channel to which the problematic frame belongs in aunit of a frame. Furthermore, in the re-sorting processing, it is alsopossible to create a new channel or combine a plurality of channels intoone channel in accordance with a user operation. That is, the re-sortingprocessing is used for manual collection of a result of the automaticsorting processing.

Upon detecting that the user has performed a predetermined operation viathe input unit 20, the automatic sorting unit 41 starts the channelseparation processing. After the start-up of the channel separationprocessing, the setting unit 42 first displays an area setting window W3for displaying a selected frame (see FIG. 5) while overlapping it withthe basic screen W1. The area setting window W3 is a screen foraccepting, from a user, the designation of an area in a selected framesuch that the area serves as a reference for calculating the similaritydegree between frames in the automatic sorting processing. “Selectedframe” refers to one frame included in a target frame group, and may be,for example, the first frame of the target frame group along thetime-line, or the active display frame. Note that although it is clearfrom the description above, the active display frame is the frame thatwas most recently selected on the time-line bar T5, and thus is sure tobe included in the target frame group.

If the target moving image is a video picture from a security camera orthe like, it is often the case that the frame screen has a date and timedisplay area (in the lower left part in the example of FIG. 5). Ifinformation of the entire frame including such a date and time displayarea is used to determine the similarity degree between frames, theremay be a risk that even frames belonging to the same channel aredetermined to have a low degree of similarity. Furthermore, blackframing may be provided in the frame screen. In such a case, conversely,if information regarding the entire frame including the black framing isused to determine the similarity degree of frames, there may be a riskthat even frames belonging to different channels are determined to havea high degree of similarity. Also in a case where the target movingimage is a mixed time-lapse moving image that is constituted by videopictures from security cameras placed in different elevators, the degreeof similarity between frames belonging to different channels is likelyto be determined as being high. This is because the scenes in theelevators are similar to each other. The area setting window W3 avoidssuch a situation, and is useful for excluding the area that may causeerroneous determination from the areas serving as a basis of thesimilarity degree calculation, so that the degree of similarity betweenframes that is to be calculated in the automatic sorting processingserves as a correct index indicating whether or not the frames belong tothe same channel. Specifically, the user designates an area thatcharacterizes the background of each channel in the selected frame ofthe area setting window W3. Taking the elevator as an example, the useronly needs to designate the area in which a poster or the like thatcharacterizes the scene of an elevator is captured. When the userperforms an operation using a mouse or the like to designate anarbitrary area in the selected frame on the area setting window W3, anenclosing line 71 that indicates the area designated by the user isdisplayed in the selected frame. Accordingly, the user can check whetheror not a correct area is designated.

Then, when the user has pressed a start button 72 on the area settingwindow W3, the automatic sorting unit 41 detects this fact and theautomatic sorting processing, which will be described later, starts.Also, at the same time, the area setting window W3 is closed. Note that,when the start button 72 has been pressed in the state in which no areais designated in the selected frame, the entire frame screen is used asa basis of the similarity degree calculation in the automatic sortingprocessing. A cancel button 73 is a button for cancelling areadesignation via the area setting window W3.

During execution of the automatic sorting processing, a dialog box forshowing the progress situation of the automatic sorting processing isdisplayed. Then, when the automatic sorting processing ends, the dialogbox is closed, and the re-sorting unit 45 starts re-sorting processing.The automatic sorting processing will be described later in detail. There-sorting unit 45 first displays a re-sorting window W4 shown in FIG.6, which serves as a user interface for performing re-sorting after theautomatic sorting processing, while overlapping it with the basic screenW1.

A channel list area C1, a frame list area C2, a play area C3, and atime-line bar C4 are arranged on the re-sorting window W4. In thechannel list area C1, channel objects 91 that respectively correspond tochannels detected in the automatic sorting processing are arranged. Inthe example of FIG. 6, four channel objects 91 that correspond to fourchannels CHO1 , CHO2, CHO3, and CHO4 are sequenced. In the presentembodiment, the channel objects 91 have an icon format.

As shown in FIG. 6, the channel objects 91 respectively indicate thechannel names CHO1, CHO2, and the like of the corresponding channels.Furthermore, each channel object 91 indicates a thumbnail image of therepresentative frame that represents a plurality of frames belonging tothe corresponding channel. The representative frame is suitably switchedin response to a user operation that will be described later, but thedefault frame is, for example, the first frame, along the time-line, ofthe frames belonging to the corresponding channel. Note that, althoughnot shown in FIG. 6, if the number of channels is so large that thechannel objects 91 corresponding to all of the channels cannot bearranged in the channel list area C1, a scroll bar will appear in thearea C1, and the area C1 will actually be expanded. Furthermore, eachchannel object 91 also indicates the number of frames belonging to thecorresponding channel. The number of frames changes in response to auser operation that will be described later, and the display of thenumber of frames in the channel object 91 will be switched in real timeis accordance with the change.

Furthermore, in addition to the channel objects 91, a non-sorted objectis also arranged in the channel list area C1. The non-sorted object 92is an object that corresponds to the sort in which non-sorted framesthat are included in the target frame group and do not belong anychannel are put together. Note that “the sort in which non-sorted framesare put together” does not correspond to an actual channel, butvirtually constitutes one channel in the sense that the frames are puttogether. Therefore, the non-sorted object 92 has a shape similar tothat of the channel objects 91, and has, in the present embodiment, anicon that has the same size as that of the channel objects 91.Specifically, the non-sorted object 92 displays “non-sorted”, in placeof the channel name, and indicates the number of non-sorted frames, inplace of the number of frames belonging to the corresponding channel.The display of the non-sorted object 92 with the number of frames isalso switched in real time in accordance with the change in the numberof non-sorted frames. Furthermore, also the non-sorted object 92displays a thumbnail image of the representative frame that representsthe “non-sorted” frames. This representative frame is also suitablyswitched in response to a user operation that will be described later,but the default frame is, for example, the first frame, along thetime-line, of the frames belonging to the sort “non-sorted”. In thepresent embodiment, a border line 93 is drawn between the non-sortedobject 92 and the channel objects 91.

The re-sorting unit 45 accepts, from a user, a selection operation forselecting an arbitrary object from among all of the channel objects 91and the non-sorted object 92 in the channel list area C1. The objectcurrently selected from among all of the objects 91 and 92 is displayedin a different mode from the other objects. In the example of FIG. 6,the object 91 of the channel CHO1 is displayed in a color different fromthe other objects 91 and 92. Hereinafter, the channel or sort thatcorresponds to the currently selected one of the objects 91 and 92 isreferred to as “selected channel”.

In the frame list area C2, thumbnail images 94 of all of the framesbelonging to the selected channel are displayed in a list. Note that ifthe number of frames belonging to the selected channel is so large thatall of the thumbnail images 94 cannot be arranged in the frame list areaC2, a scroll bar 95 will appear in the area C2, and the area C2 willactually be expanded. Accordingly, by switching the selection state ofthe objects 91 and 92, the user can selectively display, in the framelist area C2, all of the non-sorted frames or all of the framesbelonging to the corresponding channel in a list. Accordingly it ispossible to check all of the non-sorted and sorted frames included inthe target frame group while advantageously using the limited space inthe frame list area C2. Particularly, even if a wrong frame is sorted toa channel, the wrong frame will easily be found according to the listdisplay mode. Note that, in the present embodiment, the thumbnail images94 in the channel list area C1 can be sequenced in multiple stages, andare arranged in a time-series order from left to right and from upper tolower with an increase in the number n of the original frame on thetime-line.

The re-sorting unit 45 manages one specific frame belonging to eachchannel as an active frame, and also manages one specific non-sortedframe as an active frame. In the frame list area C2, an enclosing box 96is added to the thumbnail image 94 of the active frame as a mark fordistinguishing this thumbnail image 94 from the thumbnail images 94 ofthe other frames. Each thumbnail image 94 in the frame list area C2 isrecognized as an object, and the user can select, with a click operationor the like, an arbitrary thumbnail image 94 from among all of thethumbnail images 94 in the frame list area C2. Furthermore, the user canalso select a plurality of thumbnail images 94 in the frame list area C2at the same time, by repeatedly performing click operations whilepressing down a specific key on a keyboard, for example. Each time athumbnail image 94 is selected in the frame list area C2, the re-sortingunit 45 switches the active frame of the selected channel to the famethat corresponds to the most recently selected thumbnail image 94. Atthis time, the position of the enclosing box 96 also moves. Furthermore,the active frame is associated with the thumbnail images that areindicated by the channel objects 91 and the non-sorted object 92, andeach time the active frame of each channel or sort is changed, thethumbnail image of the object 91 or 92 that corresponds to the channelor sort is also switched. That is, in the present embodiment, therepresentative frame of each channel or sort corresponds to the activeframe.

The thumbnail image 94 of the active frame in the frame list area C2 canbe individually moved so as to overlap with any of the objects 91 and 92in the channel list area C1 by a drag-and-drop operation. Furthermore,if a plurality of thumbnail images 94 are selected at the same time inthe frame list area C2, the plurality of thumbnail images 94 can bemoved altogether so as to overlap with any of the objects 91 and 92 inthe channel list area C1. (Note that the thumbnail images 94 can be“moved” but if the thumbnail images 94 are released on the target object91 or 92, they will disappear as if they have been inserted into afolder. In other words, the re-sorting unit 45 accepts, from a user, anoperation for associating an arbitrary thumbnail image 94 in the framelist area C2 with an arbitrary channel or sort. After the associatingoperation, the frame that corresponds to the thumbnail image 94 that wasmoved by this operation is sorted again (re-sorted) to the channel orsort that corresponds to the object 91 or 92 to winch the thumbnailimage 94 was moved. Note that if the frame corresponding to thethumbnail image 94 as per the operation was originally sorted to thechannel or sort that corresponds to the object 91 or 92 as per theoperation, the operation will be ignored and the re-sorting will not beperformed.

As shown in FIG. 6, in the channel list area C1, a new creation object97 is arranged after the sequence of the objects 91 and 92. The newcreation object 97 is an object for creating a new channel, and has anicon format in the present embodiment. Also, the same operation as theabove-described association operation can be performed on not only theobjects 91 and 92 but also the new creation object 97. Specifically, there-sorting unit 45 accepts, from a user, an operation for independentlymoving the thumbnail image 94 of the active frame in the frame list areaC2 so that if overlaps with the new creation object 97 with adrag-and-drop operation. Furthermore, if a plurality of thumbnail images94 in the frame list area C2 are selected at the same time, theplurality of thumbnail images 94 can be moved altogether so as tooverlap with the new creation object 97. (Note that the thumbnail images94 can be “moved”, but if the thumbnail images 94 are released on thenew creation object 97, they will disappear as if they have beeninserted into a folder.) After the association operation, a channelobject 91 is newly created at the position at which the new creationobject 97 was present.

The newly created channel object 91 is an object indicating a newchannel that includes, as an element or elements, a frame or framescorresponding to one or more thumbnail images 94 moved by theassociation operation. In other words, the frame or the frames thatcorresponds or correspond to one or more thumbnail images 94 moved bythe association operation, is or are re-sorted to the new channel.Accordingly, if only one thumbnail image 94 was moved, the thumbnailimage 94 will be displayed on the object 91 immediately after thechannel object 91 is newly created. On the other hand if a plurality ofthumbnail images 94 were moved, one of these thumbnail images 94 (forexample, the most recently selected thumbnail image 94) will bedisplayed on the newly created channel object 91. Also a channel name issuitably given and displayed. Furthermore, the new creation of thechannel object 91 involves movement of the new creation object 97 behindthe sequence of the objects 91 and 92 including the newly created object91.

After the re-sorting of the frame with the above-described associationoperation, the channel name CHO1, CHO2, or the like or the label“non-sorted” is again given to the re-sorted frame, based on the channelor sort to which the frame is re-sorted. Furthermore, the thumbnailimage 94 of the re-sorted frame is deleted from the frame list area C2.Here, the remaining thumbnail images 94 are aligned in the frame listarea C2 so as to fill up the space in which the deleted thumbnail image94 was arranged. Note that the re-sorted frame or frames does or do notfail to include an active frame. Accordingly, after re-sorting, theactive frame of the selected channel is changed to the nearest one ofthe subsequent frames along the time-line, or to the nearest one of theprior frames along the time-line. The change of the active frame of theselected channel is suitably reflected by the location of the enclosingbox 96, and the indication of the thumbnail image on the object 91 or92. Furthermore, the number of frames belonging to the channel and/orsort that corresponds to the object 91 or 92 from which the re-sortedframe has been moved and to which the re-sorted frame has been moved areagain calculated, and are reflected in the display of the object 91 or92.

Furthermore, the various above-described association operations can berealized by another method than the drag-and-drop operation.Specifically, as shown in FIG. 6, character strings that indicatekeyboard-specific keys are indicated in the lower right parts of theobjects 91, 92, and 97 in the frame list area C2. Upon detecting thatthe specific key indicated in the object 91, 92, or 97 has been pressed,the re-sorting unit 45 determines that the active frame of the currentlyselected channel, that is, the frame to which the enclosing box 96 iscurrently applied in the frame list area C2 has been associated with theobject 91, 92, or 97 that corresponds to the specific key. Theprocessing thereafter is the same as in the case in which theassociation operation is performed using a drag-and-drop operation.

Furthermore, the re-sorting unit 45 can also combine a plurality ofchannels (including the sort “non-sorted” in the present and the nextparagraphs) with each other in response to a user operation.Specifically, in the channel list area C1, an arbitrary object 91 or 92can be individually moved so as to overlap with another object 91 or 92by a drag-and-drop operation. Furthermore, it is also possible that aplurality of objects 91 and 92 are selected at the same time, and aremoved altogether over another object 91 or 92. (Note that the objects 91and 92 can be “moved”, but if the objects 91 and 92 have been releasedon a destination object 91 or 92, they will disappear as if they areinserted into a folder.) In other words, the re-sorting unit 45 accepts,from a user, an operation for associating an arbitrary channel in thelist area C1 with another an arbitrary channel. After the associationoperation, the channel that corresponds to the object 91 or 92 that wasmoved by the operation is combined with the channel that corresponds tothe other object 91 or 92 to which the object 91 or 92 was moved.

After the combination of the channels with the above-describedassociation operation, the label of the channel name, such as“non-sorted”, or CHO1, CHO2, . . . , of the channel that corresponds tothe destination object 91 or 92 is again given to all of the framesbelonging to the channel that corresponds to the object 91 or 92 thatwas moved. As a result, the destination object 91 or 92 serves as anobject indicating the combined new channel. The active frame of thedestination object 91 or 92 is continuously used as the active frame ofthe combined channel. Furthermore, the re-sorting unit 45 calculates thenumber of frames belonging to the combined channel, and reflects thenumber on the indication of the object 91 or 92. Furthermore, in thechannel list area C1, if a channel object 91 is moved, the moved channelobject 91 disappears, and the remaining objects 91 and 97 are aligned soas to fill up the space in which the channel object 91, havingdisappeared, was arranged. If the non-sorted object 92 is moved, thenumber of frames is zero, but the non-sorted object 92 remains as is.

Furthermore, if there are a plurality of channel objects 91, the orderof the sequence of these objects 91 in the channel list area C1 can bechanged suitably by a drag-and-drop operation. For example, if, in thestate of FIG. 6, the channel object 91 of the channel CHO1 is attempt tobe moved to the right of the channel object 91 of the channel CHO2, thechannel object 91 of the channel CHO1 only needs to be moved between thechannel objects 91 of the channels CHO2 and CHO3.

Hereinafter, the play area C3 will be described. In the play area C3,the frame group belonging to a selected channel can be reproduced as amoving image. As shown in FIG. 6, a play button 76, a next frame button77, and a previous frame button 78 are arranged on the re-sorting windowW4.

The play button 76 accepts, from a user, an instruction to reproduce theframe group belonging to a selected channel as a moving image. Upondetecting that the play button 76 has been pressed, the re-sorting unit45 displays, in the play area C3, the frames included in the frame groupbelonging to the selected channel sequentially along the time-line in aframe-by-frame format. The reproduction starts from the active frame ofthe selected channel at a point in time when the play button 76 ispressed. Furthermore, the reproduction is performed at the frame rateaccording to the selected channel. Also, during the reproduction, theframe displayed in the play area C3 will be switched sequentially but ateach switch, the active frame managed by the re-sorting unit 45 isupdated to the switched frame in real time. Furthermore, the play button76 accepts, from a user, an instruction to stop the reproduction. Upondetecting that the play button 76 has been pressed during thereproduction, the re-sorting unit 45 fixes the display in the play areaC3 to the active frame at that point in time. Note that also whenstopped, the active frame of the selected channel is constantlydisplayed in the play area C3. Accordingly, when the active frame ischanged during the stop by the operation of selecting a thumbnail image94 in the frame list area C2, the display in the play area C3 is alsochanged in real time.

Furthermore, during the reproduction, the active frame is updated asneeded, and the enclosing box 96 in the frame list area C2 also moves tothe position to enclose the thumbnail image 94 of the most recent activeframe in real time. Furthermore, at this time, the thumbnail image ofthe object 91 or 92 that corresponds to the selected channel is updatedin real time. That is, the display in the play area C3 is synchronizedwith the position of the enclosing box 96 and the display of thethumbnail image of the object 91 or 92. However, another embodiment mayhave a configuration in which no such synchronization is made duringreproduction. In this case, it is also possible to configure such that,for example, first when reproduction is stopped, the position of theenclosing box 96 is moved, and/or, the thumbnail image of the object 91or 92 is updated.

The next frame button 77 and the previous frame button 78 respectivelyaccept, from a user, instructions to switch display in the play area C3to the next frame and to the previous frame, along the time-line, of theframe group belonging to the selected channel. The change of the displayin the play area C3 with the operation rising the buttons 77 and 78 isalso associated with the change of the active frame.

The above-described reproduction function is useful for finding a framewrongly sorted to an existing channel. That is, if a frame belonging toa different channel suddenly appears during reproduction of a movingimage, a person can immediately find the frame. In such a case, theperson only needs to immediately stop the reproduction, search for thethumbnail image 94 of the corresponding frame in the frame list area C2,and move the thumbnail image 94 to the correct channel with theabove-described association operation.

A frame rate area C5 for indicating the frame rate of a selected channelis displayed below the play area C3 on the re-sorting window W4. Theframe rate can be calculated by various methods, and can be calculatedas a value obtained by, for example, dividing the number of framesbelonging to the selected channel by a difference between the time ofthe first frame belonging to the selected channel and the time of thefinal frame. Alternatively, the frame rate of the selected channel canalso be calculated based on the following formula:

(The frame rate of the moving image corresponding to the activetime-line)×(the number of frames belonging to the selected channel)÷(thenumber of frames belonging to the active time-line)

However, as described above, the frames belonging to the selectedchannel changes with re-sorting of the frame. Accordingly, each timere-sorting is performed, the re-sorting unit 45 again calculates theframe rate of the selected channel, and changes the display in the framerate area C5 in real time. Furthermore, a channel name area C6 forindicating the channel name or sort name of a selected channel isarranged in the vicinity of the frame rate area C5. Accordingly, a usercan correctly understand to which channel the video picture displayed inthe play area C3 belongs, and to which channel the frame rate displayedin the frame rate area C5 belongs.

Hereinafter, a time-line bar C4 will be described. Similar to thetime-line bar T5 on the basic screen W1, the time-line bar C4 is anobject for schematically showing the active time-line. The target framegroup to be subjected to channel separation processing constitutes apart or whole of the active frame group. It is thus important that theactive time-line can be managed also on the re-sorting window W4, andthus the time-line bar C4 similar to the time-line bar T5 on the basicscreen W1 is arranged. The time-line bar C4 extends in the horizontaldirection, and is divided into the number of frames of the active framegroup in the horizontal direction. The n^(-th) divided region from theleft of the time-line bar C4 corresponds to the n^(-th) frame of theactive time-line (n is a natural number).

The re-sorting unit 45 displays, as a mark showing the divided regionthat corresponds to the active frame of the selected channel on thetime-line bar C4, a straight line 83 at the position of the dividedregion. That is, the straight line 83 shows the position, on the activetime-line, of the frame that is displayed in the area C3 and in theenclosing box 96 of the area C2. Furthermore, a target range bar 85 isdisplayed below the time-line bar C4. Similar to the time-line bar C4,the target range bar 85 extends in the horizontal direction over a rangethat corresponds to the segment of the target frame group.

The time-line bar C4 is extendable in the horizontal direction.Specifically, a scale changing bar 86 is arranged on the re-sortingwindow W4, and the scale changing bar 86 is an object into which twoobjects, namely, a slide groove 61 and a slider 62 are combined. Theslide groove 61 has a GUI like a groove extending straightly in thehorizontal direction, and the slider 62 has a GUI that slides in theslide groove 61 in the horizontal direction. By operating the slider 62via the input unit 20, the user can reciprocate the slider 62 along theslide groove 61 in the horizontal direction. The re-sorting unit 45changes the horizontal scale of the time-line bar C4 in a stepwisemanner, based on the position of the slider 62 in the slide groove 61 inthe horizontal direction. Specifically, the horizontal scale of thetime-line bar C4 decreases gradually with the movement of the slider 62to the right side, and the horizontal scale of the time-line bar C4increases in a stepwise manner with the movement of the slider 62 to theleft side. Note that when the horizontal scale of the time-line bar C4is changed, the sizes in the horizontal direction of all divided regionsare uniformly changed in proportion thereto. Note that, when the slider62 reaches the leftmost position in the slide groove 61, the time-linebar C4 changes to a thumbnail list C7 (see FIG. 7). Furthermore, if thehorizontal scales of the time-line bar C4 and the thumbnail list C7 aretoo long to be accommodated in the re-sorting window W4, a scroll bar 87that scrolls in the horizontal direction will appear, and the area fordisplaying the time-line bar C4 is actually extended in the horizontaldirection.

The thumbnail list C7 is a list in which thumbnail images 88 of all ofthe frames belonging to the active time-line are sequenced in thehorizontal direction. Accordingly, the user can also check, on there-sorting window W4, a frame that is included in the active time-lineand does not belong to the target frame group. The n-^(th) thumbnailimage 88 from the left on the thumbnail list C7 is the thumbnail imageof the n-^(th) frame on the active time-line (n is a natural number). Inthe thumbnail list C7, an enclosing box 89 is added to the thumbnailimage 88 of the active frame of the selected channel, the enclosing box89 serving as a mark for distinguishing this thumbnail image 88 from theother thumbnail images 88.

Similar to the time-line bar T5, the time-line bar C4 and the thumbnaillist C7 accept, from a user, the selection of an arbitrary frame on theactive time-line. In other words, the user can select a divided regionon the time-line bar C4 or an arbitrary thumbnail image 88 in thethumbnail list C7 with a click operation or the like, so as to select anarbitrary frame from among the active frame group. Also, each time aframe is selected on the time-line bar C4 and the thumbnail list C7, there-sorting unit 45 determines whether or not the frame is included inthe target frame group. If it is determined that the frame is included,the channel or sort to which the frame belongs is switched to theselected channel. At the same time, the re-sorting unit 45 switches theframe to the active frame of the selected channel. Here, the change inthe selected channel and the active frame is reflected on the display inthe channel list area C1, the frame list area C2, the play area C3, theframe rate area C5, and the channel name area C6 in real time.Furthermore, the positions of the straight line 83 and the enclosing box89 are also changed in real time.

Using the above-described various functions, the user can performre-sorting of a plurality of frames included in the target frame group.At the time when it is determined that the re-sorting has ended, an OKbutton E5 is pressed. In response thereto, the re-sorting unit 45 closesthe re-sorting window W4 and newly creates at least one moving imagebelonging to a new time-line. The at least one moving image that isnewly created has a one-to-one correspondence with the channel that wasfinally defined on the re-sorting window W4 and was selected by the user(hereinafter, referred to as “final channel”). As shown in FIG. 6, theselection of the final channel is made by the user checking a mark in aselection box 98 arranged on the channel object 91. Note that, in thepresent embodiment, the default value of the selection box 98 is“selected”. Accordingly, only with respect to the channel for which itis determined that no moving image creation is needed, the user justneeds to operate the selection box on the corresponding object 91 so asto change the state to “non-selected”. Furthermore, in the presentembodiment, a selection box 99 is also arranged below the play area C3.The selection box 99 is an object for selecting whether or not themoving image of the selected channel (note that the sort “non-sorted” isnot included therein) needs to be created, and is synchronized with theselection box 98 on the channel object 91. The re-sorting unit 45 checksthe selection states of the selection boxes 98 and 99, and decides thefinal channel.

A moving image that is newly created in the above-described manner issuch that all of the frames belonging to the corresponding final channelare sequenced along the new time-line. Also, at this time, the displaywindow W2 that has a one-to-one correspondence with the new moving imageis newly created. Furthermore, the re-sorting unit 45 creates a stillimage file group that has a one-to-one correspondence with the framegroup included in this new moving image, stores the created still imagefile group in the processed file region 52, and handles it as the0-order frame. That is, thereafter, the new moving image can beprocessed similarly to a still image file group loaded in the originalimage region 51. Specifically, the new moving image can similarly bereproduced in the display window W2, and is similarly to be subjected tovarious image processing.

Note that the moving image that corresponds to the sort “non-sorted” isnot newly created Accordingly, it is also possible that at the time whenthe OK button E5 has been pressed, the re-sorting unit 45 determineswhether or not there remains a non-sorted frame, and if it is determinedthat there remains a non-sorted frame, the user is notified of thisfact. In this case, if it is determined that there remains a non-sortedframe, the re-sorting unit 45 displays a check window W5 as shown inFIG. 8 on the display 10. When a “Yes” button 74 is pressed, theprocedure moves to the above-described processing for newly creating amoving image. When a “No” button 75 is pressed, the check window W5 isclosed, and the display returns to the re-sorting window W4. On theother hand, if it is determined that there remains no non-sorted frame,the procedure immediately moves to the above-described processing fornewly creating a moving image.

<1-4. Automatic Sorting Algorithms

Hereinafter, the automatic seating processing will be described indetail with reference to FIG. 9. As described above, the automaticsorting processing starts when the start button 72 on the area settingwindow W3 has been pressed. In the following, a plurality of framesconstituting a target frame group are denoted by F₁, F₂, . . . , F_(J)(where J is an integer of 2 or more). Note that the sequence of theframes F₁, F₂, . . . , F_(J) is equivalent to the sequence on thetime-line.

First, in step S1, the automatic sorting unit 41 scales down the size ofthe frames F₁, F₂, . . . , F_(J) included in a target frame group. Atthis time, the size of the frames may be scaled down so that the numberof the lateral and/or vertical pixels of the frames is a predeterminednumber, or so that the frames have a size that accounts for apredetermined ratio of the original size. Furthermore, at this time, anaspect ratio may or may not be saved. With this scale down step, thefollowing processing is accelerated, and the influence of noise isreduced. Hereinafter, the scaled-down frames F₁, F₂, . . . , F_(J) arealso denoted by F₁, F₂, . . . , F_(J).

In the following step S2, the setting unit 42 defines a channel CH₁, andsorts the frame F₁ to the channel CH₁. Specifically, the label channelname “CH₁” is given to the frame F₁. Furthermore, the setting unit 42sets the frame F₁ as a reference frame G_(l), which represents the framegroup that belongs to the channel CH₁. Note that by step S10, which willbe described later, the number of channels can be increased as needed.Also, for a channel CH₁ that is to be newly created a reference frameG_(l) (l=2, 3, . . . ), which, represents the frame group that belongsto the corresponding channel, will be set.

After the above-described processing has ended, the setting unit 42sequentially selects, from among the remaining frames F₂, F₃, . . .F_(J), a frame F_(j) (j=2, 3, . . . J) along the time-line, and sets theframe F_(j) as a comparative frame. Meanwhile, each time a comparativeframe F_(j) is set, the calculation unit 43, the determination unit 44,and the setting unit 42 repeatedly perform following steps S3 to S10 onthe comparative frame F_(j), and perform comparison with the existingreference frames G_(l), G₂, . . . , G_(L) (L is the number of existingchannels).

In step S3, the calculation unit 43 divides the entire comparative frameF_(j) into a predetermined number K (K is an integer of 2 or more) ofsub-regions (blocks) D₁, D₂, . . . D_(K) (see FIG. 10). Note that“entire comparative frame F_(j)” in this context refers to, if an areais designated on the area setting window W3, the entire area, whereas,if no designation was made, the entire frame screen. Note that thesub-regions D₁, D₂, . . . , D_(K) are not necessarily regions obtainedby thoroughly dividing the entire comparative frame F_(j), and maypartially overlap with each other. The shape thereof is not necessarilya rectangle, and may be, for example, a circle or another polygon.Furthermore, the sub-regions may have a uniform shape and size, ordifferent shapes and sizes.

In subsequent step S4, the calculation unit 43 divides each of thereference frames G_(l), G₂, . . . , G_(L) of the existing channels CH₁,CH₂, . . . CH_(L) into sub-regions D′₁, D′₂, . . . , D′_(K) (see FIG10). The sub-regions D′₁, D′₂, . . . , D′_(K) are respectively definedso as to be located at the same positions as those of the sub-regionsD₁, D₂, . . . , D_(K) in the frame screen.

Then, in step S5, the calculation unit 43 calculates, for eachcombination of the sub-regions D_(k) and D′_(k) (k=1, 2, . . . , K), asimilarity degree Y_(lk) (hereinafter, referred to as “local similaritydegree”) between the comparative frame F_(j) and each reference frameG_(l) (l=1, 2, . . . , L). That is, the local similarity degree Y_(lk)refers to the similarity degree between the sub-region D_(k) of thecomparative frame F_(j) and the sub-region D′_(k) of the reference frameG_(l). In the present embodiment, a correlation coefficient that iscalled “zero-mean normalized cross correlation” (ZNCC) is calculated,but in another embodiment, another similarity degree such as acorrelation coefficient that is called “NCC”, or a similarity degreethat is called “SDD” or “SAD” may be calculated.

Then, in step S6, the calculation unit 43 calculates a frame-overallsimilarity degree B₁ (hereinafter, referred to as “total similaritydegree”) between the comparative frame F_(j) and the correspondingreference frame G_(l) (l=1, 2, . . . , L). Specifically, the calculationunit 43 determines, for each l (l=1, 2, . . . , L), the sub-regionsD_(k) and D′_(k) that have a high local similarity degree Y_(lk). Atthis time, for example, the local similarity degree Y_(lk) may bedetermined as high if it is within a predetermined high order range, ormay be determined as high if it is larger than a predeterminedthreshold. Then, the calculation unit 43 again calculates, for each l(l=1, 2, . . . , L), the similarity degree between the comparative frameF_(j) and the reference frame G_(l) only using the sub-regions D_(k) andD′_(k) that have a high local similarity degree Y_(lk) in the entireframe, and sets the calculated similarity degree as the total similaritydegree B₁ (see FIG. 10). In the present embodiment, a zero-meannormalized cross correlation (ZNCC) is also calculated as the totalsimilarity degree B_(t), but another similarity degree may becalculated, similar to the case of the local similarity degree Y_(lk).Furthermore, the total similarity degree B_(t) and the local similaritydegree Y_(lk) may be calculated using different methods.

Accordingly, in the present embodiment, the information relating to thesub-regions D_(k) and D′_(k) that have a low local similarity degreeY_(lk) is not taken into consideration for the calculation of the totalsimilarity degree B₁, and the reason thereof is as follows. That is,frames belonging to the same channel have the same background imageexcept for a moving subject part. If the moving subject part accountsfor a large ratio in the entire frame, and the total similarity degreeB₁ is determined using the information relating to the entire frame,even frames belonging to the same channel will have a low totalsimilarity degree B₁. Accordingly, here, the sub-regions D_(k) andD′_(k) that are considered to include a large moving subject part andhave a low local similarity degree Y_(lk) are not used in thecalculation of the total similarity degree B₁, in order to preventerroneous determination. As a result, the influence of a moving subjectis reduced, and it is possible to correctly determine whether or notcompared frames belong to the same channel.

In subsequent step S7, the determination unit 44 determines the maximumof the total similarity degrees B₁, B₂, . . . , B_(L) calculated in stepS6, and determines whether or not the maximum total similarity degree islarger than a predetermined threshold. If it is determined that themaximum total similarity degree is larger than the predeterminedthreshold, the procedure advances to step S8, whereas if it isdetermined that the maximum total similarity degree is the predeterminedthreshold or less, the procedure advances to step S10.

In step S8, the determination unit 44 determines that the comparativeframe F_(j) belongs to a channel CH_(MAX). “MAX” is a value of “1” ofthe maximum total similarity degree B₁. Specifically, the determinationunit 44 gives the channel name label “CH_(MAX)” to the comparative frameF_(j), and sorts the comparative frame F_(j) to the channel CH_(MAX).Accordingly, the comparative frame F_(j) will belong to the same channelas that of the frame group belonging to the channel CH_(MAX).

After step S8, the procedure advances to step S9. In step S9, thesetting unit 42 updates a reference frame G_(MAX) of the channelCH_(MAX). Specifically, the setting unit 42 combines the comparativeframe F_(j) with the existing reference frame G_(MAX) into a newreference frame G_(MAX). In the present embodiment, weighted averagingis employed as a combination method, in which a larger weight is givento the comparative frame F_(j) than the existing reference frameG_(MAX). Accordingly the reference frame G_(MAX) that represents theframe group belonging to the channel CH_(MAX) is a combined image inwhich, a larger weight is given to a nearer frame. As a result, whendetermining whether or not frames belong to different channels, it ispossible to support a time-series change. Note that in otherembodiments, the comparative frame F_(j) may directly be set as a newreference frame G_(MAX). Alternatively, step S9 may be omitted so thatan update of the reference frame G_(MAX) is not performed.

On the other hand, step S10 is a step that is executed when all of thetotal similarity degrees B₁, B₂, . . . B_(L) calculated in step S6 aredetermined as being a predetermined threshold or less. In other words,step S10 is a step that is executed when the comparative frame F_(j) isnot similar to the reference frame G_(l) of any of the channels CH₁. Instep S10, the setting unit 42 defines a new channel CH_(L+1), inaddition to the existing channels CH₁, CH₂, . . . , CH_(L), and sortsthe comparative frame F_(j) to the channel CH_(L+1). Specifically, a newchannel name label “CH_(L+1)” is given to the comparative frame F_(j).Furthermore, the setting unit 42 sets the comparative frame F_(j) as areference frame G_(L+1), which represents the frame group belonging tothe new channel CH_(L+1).

When steps S3 to S10 on all of the frames F₁, F₂, . . . F_(J) haveended, the procedure advances to step S11. In step S11, the existingchannels CH₁, CH₂, . . . , CH_(L) are corrected. Specifically, thecalculation unit 43 calculates the frame-overall similarity degreebetween the reference frames G_(l), G₂, . . . , G_(L) in an all-to-allmanner. Note that, similar to step S3, “frame-overall” in this contextrefers to, if an area is designated on the area setting window W3, theentire area, whereas, if no designation was made, the entire framescreen. Furthermore, in the present embodiment, a zero-mean normalizedcross correlation (ZNCC) is calculated as the similarity degree butanother similarity degree may be calculated, similar to steps S5 and S6.Furthermore, the similarity degree may be calculated by a methoddifferent from methods by which the total similarity degree B₁ and thelocal similarity degree Y_(lk) are calculated. Also, if such asimilarity degree exceeds a predetermined threshold, the determinationunit 44 combines the channels CH₁ that correspond to the referenceframes G_(l) that give such a similarity degree into one channel.Specifically, the same channel name label is again given to all of theframes F_(j) belonging to the channels CH₁ to be combined. Note herethat three or more channels CH₁ may be combined into one.

Furthermore, if there is a channel CH₁ that includes only one frame evenafter the above-described combination of channels, the determinationunit 44 deletes the channel CH₁. Specifically, the determination unit 44again gives the label “non-sorted” to all of the frames F_(j) belongingto the channel CH₁ to be deleted. Note that, in the present embodiment,the channel CH₁ that includes only one frame F_(j) is to be deleted, butsuch a reference value may be two, three, or the like. Furthermore, itis also possible to set an upper limit for the number of channels thatcan be created. In this case, deletion can be started from the channelthat includes the smallest number of frames F_(j), so that the number ofchannels does not exceed the upper limit.

According to the above-described processing, it is likely that thevalues included in the channel names CH₁ of the finally created channelsare not a consecutive number. Accordingly, the determination unit 44again gives the channel names such as CH₀₁, CH₀₂, and the likesequentially to the finally created channels CH₁, and updates the labelsgiven to the frames F_(j) to these new channel names. Then, theautomatic sorting processing ends.

2. Second Embodiment

Hereinafter, a second embodiment of the present invention will bedescribed. FIG. 11 shows a flow of automatic sorting processingaccording to the second embodiment, and the second embodiment differsfrom the first embodiment in an automatic sorting algorithm.Furthermore, as is clear from the comparison between FIG. 9 and FIG. 11,a difference between both of the automatic sorting processes is that,instead of steps S4 to S6, steps S24 to S26 are inserted in the secondembodiment. Accordingly, in the following, for case of description, onlythe difference will be described.

In the second embodiment, after execution of steps S1 to S3 as with thefirst embodiment, step S24 is executed. In step S24, the calculationunit 43 detects feature points in each of the sub-regions D₁, D₂, . . ., D_(K) of the comparative frame F_(j). At that time, the numbers of thefeature points that are detected in the respective sub-regions D₁, D₂, .. . , D_(K) are preferably the same. Hereinafter, the feature points ofthe comparative frame F_(j) are denoted as P₁, P₂, . . . , P_(U) (U isan integer of 2 or more).

The calculation unit 43 sets, in the comparative frame F_(j), asub-region V_(u) in the vicinity of each feature point P_(u) (u=1, 2, .. . , U) (see FIG. 12). The sub-region V_(u) is a region of apredetermined size in which the feature point P_(u) is located at thecenter. Then, the calculation unit 43 sets sub-regions V′₁, V′₂, . . . ,V′_(U) for each of the reference frames G_(l), G₂, . . . , G_(L) of theexisting channels CH₁, CH₂, . . . , CH_(L) (see FIG. 12). Thesub-regions V′₁, V′₂, . . . , V′_(U) are respectively defined so as tobe located at the same positions as those of the sub-regions V₁, V₂, . .. , V_(U) in the frame screen.

As described above, in the present embodiment the feature points P₁, P₂,. . . , P_(U) are detected from the comparative frame F_(j) on a basisof the sub-regions D₁, D₂, . . . , D_(K). As a result, the featurepoints P₁, P₂, . . . , P_(U) are detected substantially uniformly fromthe entire comparative frame F_(j) without being biased to a partialregion of the entire screen of the comparative frame F_(j).

Then, in step S26, the calculation unit 43 calculates, for eachcombination of sub-regions V_(u) and V′_(u) (u=1, 2, . . . , L), thelocal similarity degree Y_(lk) between the comparative frame F_(j) andeach reference frame G_(l) (l=1, 2, . . . , L). That is, “localsimilarity degree Y_(lk)” refers to the similarity degree between thesub-region V_(u) of the comparative frame F_(j) and the sub-regionV′_(u) of the reference frame G_(l). Note that the local similaritydegrees in the first and second embodiments are common in that theydenote the similarity degrees between sub-regions, and thus the samereference sign Y is used to denote the local similarity degrees.Furthermore, in the present embodiment, a correlation coefficient,called a zero-mean normalized cross correlation (ZNCC), is calculated,but another similarity degree may be calculated, similar to the firstembodiment.

Then, in step S26, the calculation unit 43 calculates the frame-overalltotal similarity degree B₁ between the comparative frame F_(j) and thecorresponding reference frame G_(l) (l=1, 2, . . . , L). Note that thetotal similarity degrees in the first and second embodiments are commonin that they denote the similarity degrees in the entire frame, and thusthe same reference sign B is used for denoting the total similaritydegrees. Specifically, the calculation unit 43 determines, for each l(l=1, 2, . . . , L), the sub-regions V_(u) and V′_(u) that have a highlocal similarity degree Y_(lu). At this time, for example, the localsimilarity degree may be determined as high if the local similaritydegree Y_(lu) is in a predetermined high order range, or may bedetermined as high if the local similarity degree Y_(lu) is larger thana predetermined threshold. Then, for each l (l=1, 2, . . . , L), thesimilarity degree between the comparative frame F_(j) and the referenceframe G_(l) is again calculated only using the sub-regions V_(u) andV′_(u) of the entire frame that have a high local similarity degreeY_(lu), and the calculated similarity degree is set as the totalsimilarity degree B₁. Also in the present embodiment, a zero-meannormalized cross correlation (ZNCC) is calculated as the totalsimilarity degree B₁, but another similarity degree may be calculatedsimilar to the first embodiment. Also in the present embodiment, thetotal similarity degree B₁ and the local similarity degree may becalculated by different methods.

With step S26 above, the total similarity degrees B₁, B₂, . . . , B_(L)are calculated. The flow of the processing thereafter is the same as inthe first embodiment.

As described above, in the present embodiment, information relating toregions other than the sub-regions V₁, V₂, . . . V_(U) in the vicinityof the feature points P₁, P₂, . . . P_(U) is not taken intoconsideration for the calculation of the total similarity degrees. Thisis because, if frames, even of different channels, are of imagesobtained by capturing similar scenes, and information relating to theentire frame is used for determining the total similarity degree B₁, ahigh total similarity degree B₁ will be obtained. For example, in a caseof two video pictures in which different places of the same shopinterior are captured, the major part of the frame of both videopictures may include wallpaper, a floor, and the like in the sameuniform color, and in such a case, the total similarity degree B₁ ishigh. Accordingly, here, the regions in which a background such aswallpaper and a floor in the same uniform color is likely to becaptured, that is, the regions that are not located in the vicinity ofthe feature points P₁, P₂, . . . , P_(U) are not used in the calculationof the total similarity degree B₁, in order to prevent erroneousdetermination. As a result, it is possible to reduce the influence of asimilar background, and to correctly determine whether or not comparedframes belong to the same channel.

Furthermore, in the present embodiment, information relating to thesub-regions V_(lu) and V′_(u) that have a low local similarity degreeY_(lu) is also not taken into consideration for the calculation of thetotal similarity degree B₁. This is because, if information relating tothe entire frame is used to determine the total similarity degree B₁when the ratio of a moving subject part in the entire frame is large,frames, even of the same channel, will have a low total similaritydegree B₁. Accordingly, here, the sub-regions V_(u) and V′_(u) that havea low local similarity degree Y_(lu) and that are considered to includea large moving subject part are not used for the calculation of thetotal similarity degree B₁, in order to prevent erroneous determination.As a result, it is possible to reduce the influence of the movingsubject, and to more correctly determine whether or not compared framesbelong to the same channel.

3. Third Embodiment

Hereinafter, a third embodiment according to the present invention willbe described. FIG. 13 shows a flow of automatic sorting processingaccording to the third embodiment and the third embodiment differs fromthe first and second embodiments only in an automatic sorting algorithm.Furthermore, as is clear from the comparison between FIG. 13 and FIG. 9,a difference between automatic sorting processes according to the firstand third embodiments is only that, instead of steps S5 and S6, subroutine S30 is inserted in the third embodiment. Accordingly, in thefollowing, for case of description, only the difference will bedescribed.

As described above, in the first and second embodiments, a correlationcoefficient is calculated, as a similarity degree between frames, but inthe third embodiment, the similarity degree between frames is evaluatedbased on color trends of the frames. Furthermore, as the color trend ofa frame, an overall color trend in the screen of the frame and a localcolor trend in the screen of the frame are comprehensively taken intoconsideration. In the following, a specific description will be given.

First, in the third embodiment, steps S1 to S4 as with the firstembodiment are executed, and then sub routine S30 is executed. Subroutine S30 is processing for evaluating various color indexesindicating local or overall color trends in the frame screen, and isshown in detail in FIG. 17. Particularly, density is considered in stepsS31 to S33 included in sub routine S30, chroma is considered in stepsS34 to S38, and hue is considered in steps S39 to S43. Note thatalthough density is considered in steps S31 to S33, intensity orbrightness may be used instead of density. Furthermore, frame-overallimpression is considered in steps S44 to 46 included in sub routine S30.

First, in step S31, the calculation unit 43 calculates an averagedensity Hd of the entire comparative frame F_(j). Note that “entirecomparative frame F_(j)” in this context refers to, if an area isdesignated on the area setting window W3, the entire area, whereas, ifno designation was made, the entire frame screen. Furthermore, thecalculation unit 43 calculates an average density Ed_(k) of eachsub-region D_(k) (k=1, 2, . . . ., K) of the comparative frame F_(j).Note that “average density” is calculated as an average of the densitiesof all pixels in the target area, and the density of each pixel iscalculated as an average of RGB values of the pixel. Then, thecalculation unit 43 gives the value “1” to each sub-region D_(k) (k=1,2, . . . , K) if density Ed_(k)>density Hd is satisfied, and gives thevalue “0” if density Ed_(k)≦density Hd is satisfied (see FIG 14). Notethat the values “1” and “0” are indexes that indicate whether thedensity of a local region is relatively high or low relative to theentire frame, and are color indexes that indicate the local color trendregarding density (brightness) (hereinafter, referred to as “densityindex”).

In subsequent step S32, the calculation unit 43 executes the sameprocessing as step S31 on each of the reference frames G_(l), G₂, . . ., G_(L) of the existing channels CH₁, CH₂, CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), adensity index with respect to each sub-region D′_(k) (k=1, 2, . . . , K)in the reference frame G_(l) (see FIG. 14). Note that, as shown in FIG.13 and FIG. 17, step S32 will be repeatedly executed, and thus, for thereference frame G_(l) for which the density index has been calculated,this value is referenced and new calculation processing is omitted.

In subsequent step S33, the calculation unit 43 calculates, based on thedensity indexes calculated in steps S31 and S32, a similarity degree Bd₁between the comparative frame F_(j) and each reference frame G_(l) (l=1,2, . . . , L). Specifically, the calculation unit 43 calculates, as thesimilarity degree Bd₁, a similarity degree between a density indexdistribution pattern in the comparative frame F_(j) and a density indexdistribution pattern of each reference frame G_(l). In the presentembodiment, the number of combinations of the sub-regions D_(k) andD′_(k) whose density indexes “1” and “0” do not match is counted, andthe square value of the value obtained by dividing tins count value by asub-region count K is set as the similarity degree Bd₁ (see FIG. 14).Note here that the reason why division using the sub-region count K isperformed is to maintain the similarity degree Bd₁ as a value between 0and 1, and to normalize it. Furthermore, the smaller the similaritydegrees calculated in steps S33, and steps S38 and S43, which will bedescribed later, the higher the similarity is. In this sense, thesesimilarity degrees can be handled as “non-similarity degrees”.

Meanwhile, a composition of an image is decided by arrangement of aplurality of elements in the screen. Many images, if simplified, have acomposition in which a region showing an attention object and a regionshowing the background thereof are broadly separated. That is, theregion having the density index “1” and the region having the densityindex “0” can be considered such that one shows an attention object andthe other shows the background thereof. Based on this consideration, itcan be said that the density index distribution pattern in thecomparative frame F_(j) refers to the composition of the comparativeframe F_(j), and the density index distribution pattern in the referenceflame G_(l) refers to the composition of the reference frame G_(l).Accordingly, it can be said that the similarity degree Bd₁ refers to thesimilarity degree of the composition between the comparative frame F_(j)and the reference fame G_(l).

In subsequent step S34, the calculation unit 43 divides, based on thedensity index distribution pattern in the comparative frame F_(j), thecomparative frame F_(j) into two sub-regions R₁ and R₂ (see FIG. 15).The sub-regions R₁ are regions obtained by combining all of thesub-regions D_(k) whose density index is “1” in the comparative frameF_(j), and the sub-regions R₂ are regions obtained by combining all ofthe sub-regions D_(k) whose density index is “0” in the comparativeframe F_(j). That is, in step S34, the comparative frame F_(j) isdivided into the region showing the attention object and the regionshowing the background thereof, based on the composition of thecomparative frame F_(j).

In subsequent step S35, similar to step S34, the calculation unit 43divides, based on the density index distribution pattern of each of thereference frame G_(l), G₂, . . . , G_(L) of the existing channels CH₁,CH₂, . . . , CH_(L), each of the reference frame G_(l), G₂, . . . ,G_(L) into sub-regions R′₁ and R′₂ (see FIG. 15). The sub-region R′₁ isa region obtained by combining all of the sub-regions D′_(k) whosedensity index is “1” in the reference frame G_(l), and the sub-regionR′₂ is a region obtained by combining all of the sub-regions D′_(k)whose density index is “0” in the reference frame G_(l). That is, instep S35, the reference frame G_(l) is divided into the region showingan attention object and the region showing the background thereof basedon the composition of the reference frame G_(l).

In subsequent step S36, the calculation unit 43 calculates, for each ofthe sub-regions R₁ and R₂ in the comparative frame F_(j), three averagesfor each of RGB. Then, the calculation unit 43 calculates a chroma ofthe sub-region R₁ based on the thee averages for each of RGB in thesub-region R₁, and calculates a chroma of the sub-region R₂ based on thethree averages for each of RGB in the sub-region R₂. Then, thecalculation unit 43 calculates a chroma that is relative between thesub-regions R₁ and R₂ (hereinafter, referred to as relative chroma). Therelative chroma is calculated as the absolute value of a difference inchroma between the sub-region R₁ and the sub-region R₂. The relativechroma is a color index indicating the color trend regarding chroma(hereinafter, referred to as chroma index).

In subsequent step S37, the calculation unit 43 executes the sameprocessing as step S36 on each of the reference frames G_(l), G₂, . . .G_(L) of the existing channels CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), a chromaof each of the sub-regions R′₁ and R′₂ in the reference frame G_(l), andcalculates the relative chroma, which is the absolute value of adifference between these. Note that, as shown in FIG. 13 and FIG. 17,step S37 will be repeatedly executed, and thus, for the reference frameG_(l) for which the relative chroma has been calculated, this value isreferenced and new calculation processing is omitted. Step S35 may alsobe omitted similarly.

In subsequent step S38, the calculation unit 43 calculates a similaritydegree Bs₁ between the relative chroma of the comparative frame F_(j)and the relative chroma of each reference frame G_(l) (l=1, 2, . . . ,L), which were calculated in steps S36 and S37. In the presentembodiment. The similarity degree Bs₁ is calculated as the square valueof the value obtained by dividing a difference between both relativechromas by 255. Note here that the reason why division using 255 isperformed is to maintain the similarity degree Bs₁ between 0 and 1, andto normalize it

In subsequent step S39, the calculation unit 43 divides the comparativeframe F_(j) into two regions, namely, a main region O₁ and a sub regionO₂. Specifically, of the sub-regions R₁ and R₂ calculated in step S34,the region having a larger area is set as the main region O₁, and theregion having a smaller area is set as the sub region O₂ (see FIG. 16).That is, in step S39, the comparative frame F_(j) is divided into theregion showing an attention object and the region showing the backgroundthereof, based on the composition of the comparative frame F_(j).

In subsequent step S40, similar to step S39, the calculation unit 43divides each of the reference frames G_(l), G₂, . . . , G_(L) into tworegions, namely, a main region O′₁ and a sub region O′₂. Specifically,of the sub-regions R′₁ and RS calculated in step S35, the region havinga larger area is set as the main region O′₁, and the region having asmaller area is set as the sub region O′₂ (see FIG. 16). That is, instep S40, the reference frame G_(l) is divided into the region showingan attention object and the region showing the background thereof, basedon the composition of the reference frame G_(j).

In subsequent step S41, the calculation unit 43 calculates an averagehue of the main region O₁, and calculates an average hue of the subregion O₂. Note that these average hues are each color index indicatinga local color trend regarding hue (hereinafter, referred to as “hueindex”), and ate each calculated as an average of hues of all pixels inthe target area.

In subsequent step S42, the calculation unit 43 executes the sameprocessing as step S41 on each of the reference frames G_(l), G₂, . . ., GL of the existing channels CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), a hueindex of each of the regions O′₁ and O′₂ in the reference frame G_(l).Note that, as shown in FIG. 13 and FIG. 17, step S42 will be repeatedlyexecuted, and thus, for the reference frame G_(l) for which the hueindex has been calculated, this value is referenced and new calculationprocessing is omitted. Step S40 may also be omitted similarly.

In subsequent step S43, the calculation unit 43 calculates a similaritydegree Bh_(u) between the hue index of the main region O₁ in thecomparative frame F_(j), and the hue index of the main region O′₁ ineach reference frame G_(l) (l=1, 2, . . . , L). In the presentembodiment, the similarity degree Bh_(u) is calculated as the squarevalue of the value obtained by dividing the difference between the hueindexes of the main regions O₁ and O′₁ by 180. Furthermore, thecalculation unit 43 calculates a similarity degree Bh₂₁ between the hueindex of the sub region O₂ in the comparative frame F_(i), and the hueindex of the sub region O′₂ in each reference frame G_(l) (l=1, 2, . . ., L). Specifically, the square value of the value obtained by dividingthe difference between the hue indexes of the sub regions O₂ and O′₂ by180 is set as the similarity degree Bh₂₁. Note here that the reason whydivision using 180 is performed is to maintain the similarity degreesBh₁₁ and Bh₂₁ between 0 and 1, and to normalize them.

Subsequent steps S44 to S46 are steps for evaluating a color indexindicating an overall color trend in the frame screen. In the presentembodiment, various impressions Z₁, Z₂, . . . , Z_(I) (I is an integerof 2 or more) are defined in order to evaluate the overall color trendin the frame screen. Note that various types of information for definingthese impressions Z₁, Z₂, . . ., Z_(I) are stored in the impressiondefining region 53 of the software management region 50.

Hereinafter, the impression Z_(i) (i=1, 2, . . . , I) will be described.Each impression Z_(i) is associated with one or more color conditions,and weights are defined for the color conditions. Note that the weightsof one or more color conditions that are associated with the sameimpression Z_(i) is summed up to 1. FIG. 18 shows an example of theimpression Z₁ “natural”, and the impression Z₁ “natural” is associatedwith three color conditions “green”, “brown”, and “beige”. Furthermore,the weights 0.5, 0.25., 0.25 are respectively given to the three colorconditions “green”, “brown”, and “beige”. As shown in FIG. 18, for eachof the color conditions, evaluation values for the values of density(intensity), chroma, and hue are defined. Here, in a case of a pixel ofinterest, by obtaining evaluation values for the values of density,chroma, and hue of the pixel and then multiplying these evaluationvalues by each other, the value of each color condition is calculated.Also, the value of the impression Z_(i) of the pixel is calculated as avalue obtained by giving the above-described weights to the values ofthe color conditions and multiplying these by each other.

In step S44, the calculation unit 43 calculates the value of eachframe-overall impression Z_(i) (i=1, 2, . . . , I) of the comparativeframe F_(j). Note that “frame-overall” in this context refers to, if anarea is designated on the area setting window W3, the entire area,whereas, if no designation was made, the entire frame screen.Specifically, the calculation unit 43 calculates, for each i (i=1, 2, .. . , I), the value of the impression Z_(i) for each pixel included inthe entire comparative frame F_(j), calculates an average of thesevalues, and sets the average as the value of the frame-overallimpression Z_(j). Note that the value of the frame-overall impressionZ_(i) is a color index indicating a frame-overall color trend regardingthe impression Z_(i) (hereinafter, referred to as “impression index”).

In subsequent step S45, the calculation unit 43 executes the sameprocessing as step S44 on each of the reference frames G_(l), G₂, . . ., G_(L) of the existing channel CH₁, CH₂, . . . , CH_(L). That is, thecalculation unit 43 calculates, for each l (l=1, 2, . . . , L), animpression index of the reference frame G_(l) with respect to each ofthe impressions Z₁, Z₂, . . . , Z_(I). Note that, as shown in FIG. 13and FIG. 17, step S45 is repeatedly executed, and thus, for thereference frame G_(l) for which the impression index has beencalculated, this value is referenced and new calculation processing isomitted.

In subsequent step S46, the calculation unit 43 calculates aframe-overall similarity degree Bi_(l) between the comparative frameF_(j) and each reference frame G_(l) (l=1, 2, . . . , L), based on theimpressions Z₁, Z₂, . . . , Z₁. Specifically, the calculation unit 43calculates, for each l (l=1, 2 . . . , L) and each i (i=1, 2, . . . ,I), the square value of the difference between the value of fileframe-overall impression Z_(i) of the comparative frame F_(j) and theframe-overall impression Z_(i) of the reference frame G_(l). Then, foreach l (l=1, 2, . . . , L), the calculation unit 43 obtains a value bysubtracting the square root of the sum of I square values (the distancebetween the comparative frame F, and the reference frame G_(l) in theI-order impression space) from 1, and sets the resulting value as thesimilarity degree Bi_(l).

Then, in step S47, the calculation unit 43 calculates, based on thealready calculated similarity degrees Bd₁, Bs₁, Bh₁₁, Bh₂₁, and Bi_(l)(l=1, 2, . . . , L), a similarity degree B₁ between the comparativeframe F_(j) and each reference frame G_(l) (l=1, 2, . . . , L)(hereinafter, referred to as “total similarity degree”). Note that thetotal similarity degrees in the first and third embodiments are commonin that they denote the similarity degrees in the entire frame, and thusthe same reference sign B is used for denoting the festal similaritydegrees. In the present embodiment, the calculation unit 43 calculates,for each l (l=1, 2, . . . , L), a value (composition comparison result)that is obtained by subtracting the square root of the sum of Bd₁, Bs₁,Bh₁₁, Bh₂₁ (the distance between the comparative frame F_(j) and thereference frame G_(l) in the 4^(th)-order space relating to fourindexes) from 1, and then sets a value obtained by multiplying thisvalue by Bj₁ (impression comparison result) as the total similaritydegree B₁. Note that a different method may also be employed in otherembodiments, and a value may be obtained, for example, by multiplying asuitable coefficient by the composition comparison result and/or theimpression comparison result, or adding or subtracting a suitable valueto or from the composition comparison result and/or the impressioncomparison result, and then multiplying the results by each other.Alternatively, a value obtained by summing up the square root of 1−Bd₁,the square root of 1−Bs₁, the square root of 1−Bh₁₁, the square root of1−Bh₂₁, and Bi_(l) while giving a suitable coefficient thereto may beset as the total similarity degree B₁.

With step S47 above, the total similarity degrees B₁, B₂, . . . , B_(L)are calculated. The flow of processing thereafter is the same as in thefirst embodiment.

4. Usage

The image processing program 2 can handle image processing with respectto various types of moving images, and can be used also in a scenewhere, for example, an organization such as the police analyzessurveillance images from security cameras in order to investigate anincidents For example, there may the a case where many security camerasare placed in the same shop, and these surveillance images are oftenrecorded in the format of a mixed time-lapse moving image. In such acase, dedicated equipment in the shop in which the security cameras areplaced can be used to separate channels included in the mixed time-lapsemoving image, but in a police facility, which does not always possessthe dedicated equipment that supports devices of manufacturers,reproduction is difficult. Accordingly, the above-described time-lineseparation processing in which a mixed time-lapse moving image can beseparated into moving images of channels regardless of thespecifications of a moving image can be used in such a scene.

5. Modification

Although the embodiments of the present invention have been described sofar, the present invention is not limited to the foregoing embodimentsand various modifications are possible without departing from the spiritof the invention. For example, the following modifications are possible.

<5-1>

The re-sorting window W4 of the foregoing embodiments may be constitutedby a plurality of windows.

<5-2>

In the foregoing embodiments, by switching the selected channel, thedisplay of a frame group in the frame list area C2 can be switched on achannel or sort basis. However, a frame list area for displaying a framegroup in a list may be provided for each of multiple channels and thesort “non-sorted”. In this case, in order to efficiently use a screenspace, thumbnail display for the frame group is preferably performed inthe frame list area, and also a thumbnail image group is preferablysequenced along the time-line. Furthermore, in this case, the channelobjects 91 and the non-sorted object 92 that have an icon format can beomitted, and the frame list area for each channel can be used as achannel object for the association operation, and the frame list areafor the sort “non-sorted” can be used as a non-sorted object for theassociation operation. That is, a configuration is also possible inwhich an arbitrary frame is selected from the “non-sorted” frame listarea, and is dragged and dropped to an arbitrary channel frame list area(channel object), so as to be able to be sorted to the channelcorresponding to the area. The same applies to the case where a frameincluded in a channel is moved to another channel or sort.

Alternatively, a frame list area for displaying the “non-sorted” framegroup in a list and a frame list area for displaying the frame groupbelonging to a channel in a list may be provided, so that the“non-sorted” frame group is constantly displayed in the former area. Inthis case, in the latter area, only the frame group belonging to thecurrently selected channel can be displayed in a list

<5-3>

In the third embodiment, various color indexes are defined, and arecombined in various manner so as to define the total similarity degreeB₁, but it is also possible to define the color indexes with anothermethod, or combine the color indexes with another method to define thetotal similarity degree B₁. For example, in the third embodiment acomposition is decided based on the color index regarding density, butthe composition may be decided based on the color index regarding hueand/or chroma. Furthermore, the similarity degree Bs₁ is calculatedbased on the relative chroma between the sub-regions R₁ and R₂, but adifference in absolute chroma between the sub-regions R₁ and R′₁, or adifference in absolute chroma between the sub-regions R₂ and R′₂ may beset as the similarity degree Bs₁. Furthermore, a difference in densitybetween the sub-regions R₁ and R′₁, or a difference in density betweenthe sub-regions R₂ and R′₂ may be calculated as a similarity degree, andmay be added to the total similarity degree B₁.

<5-4>

The foregoing embodiments have a configuration in which the area settingwindow W3 is displayed immediately after the channel separationprocessing has started, so that area setting for preventing erroneousdetermination is possible. However, it is also possible that this areasetting window W3 is not displayed so that such area setting isimpossible. Alternatively, a configuration is also possible in which,although the area setting window W3 is not displayed immediately afterthe channel separation processing has started, the re-sorting window W4has a button for giving a command to restart the automatic sortingprocessing after the area setting has been performed. In this case, onlywhen a result of the automatic sorting processing that was performedwithout performing area setting (the result includes a lot of erroneousdeterminations) is inappropriate, a user presses this button anddisplays the area setting window W3. Then, area setting is performed,and then the automatic sorting processing can again be executed.Alternatively, a configuration is also possible in which the areasetting window W3 is displayed immediately after the channel separationprocessing has started, so that area setting is performed, and theautomatic sorting processing can be executed in this state, andfurthermore, the above-described button is also provided on there-sorting window W4.

<5-5>

In step S9 of the foregoing embodiments, the combination of thecomparative frame F_(i) and the existing reference frame G_(MAX) usingweighted averaging is set as a new reference frame G_(MAX). However, themethod for updating the reference frame G_(MAX) is not limited to this.For example, for each pixel, a histogram of pixel values of all frames(also including the comparative frame F_(j) at this point in time)included in the channel CH_(MAX) may be created, and the mode pixelvalue may be set as the pixel value of this pixel of the reference frameG_(MAX). Alternatively, the pixel value serving as the central value ofthe histogram may be set as the pixel value of this pixel of thereference frame G_(MAX).

<5-6>

In the foregoing embodiments, a configuration can be employed in whichimmediately after step S10 or the like ends, it is determined whether ornot a channel count L has reached a predetermined number, and if it isdetermined that the channel count L has reached the predeterminednumber, processing for correcting an existing channel as with in stepS11 is performed. This is because, if the channel count L increases toomuch, the processing load becomes very large. Note that it is preferableto employ a configuration such that, if there is a channel whose framecount is one, and a temporal distance from this one frame to thecurrently selected comparative frame F_(j) is within a predeterminedrange, the channel is not deleted. This is because this channel is arecently created channel, and a frame to be sorted to this channel islikely to appear in the future. Note that, if even only performing theabove-described channel deletion results in the channel count Lexceeding the predetermined value, it is also possible to sequentiallyset the channels whose count is 2 or more, 3 or more or the like astargets of deletion, and to keep the channel count equal to or smallerthan the predetermined number.

REFERENCE SIGNS LIST

1 Image processing device

2 Image processing program

41 Automatic sorting unit

42 Setting unit

43 Calculation unit

44 Determination unit

45 Re-sorting unit

91 Channel objects

92 Non-sorted object

C3 Play area

W4 Re-sorting window (re-sorting screen)

1. An image processing device for sorting frames included in one movingimage into different channels, the frames included in the moving imagebeing a mixture of frames belonging to the different channels,comprising an automatic seating unit configured to calculate asimilarity degree of the plurality of frames included in the movingimage by performing image processing on the frames, and sort theplurality of frames into the plurality of channels based on thesimilarity degree.
 2. The image processing device according to claim 1,wherein the automatic sorting unit includes: a setting unit configuredto set a specific frame included in The moving image or a frame that isobtained by combining two or more specific frames included in the movingimage as a reference frame, and set another specific frame included inthe moving image as a comparative frame; a calculation unit configuredto execute calculation processing of dividing the comparative frame andthe reference frame into a plurality of sub-regions, calculating a localsimilarity degree between the comparative frame and the reference framefor each sub-region, and calculating a total similarity degree, which isa frame-overall similarity degree, between the comparative frame and thereference frame only using sub-regions having a high local similaritydegree; and a determination unit configured to execute determinationprocessing of determining, based on the total similarity degree, whetheror not the comparative frame belongs to the same channel as that of thereference frame or the frames combined into the reference frame.
 3. Theimage processing device according to claim 1, wherein the automaticsorting unit includes: a setting unit configured to set a specific frameincluded in The moving image or a frame that is obtained by combiningtwo or more specific frames included in the moving image as a referenceframe, and set another specific frame included in the moving image as acomparative frame; a calculation unit configured to perform calculationprocessing of detecting feature points from the comparative frame,setting a vicinity region for each feature point, and calculating atotal similarity degree, which is a frame-overall similarity degree,between the comparative frame and the reference frame only using thevicinity regions; and a determination unit configured to executedetermination processing of determining, based on the total similaritydegree, whether or not the comparative frame belongs to the same channelas that of the reference frame or the frames combined into the referenceframe.
 4. The image processing device according to claim 3, wherein thecalculation unit is configured to divide the comparative frame into aplurality of sub-regions, and detect the feature points for eachsub-region from the comparative frame.
 5. The image processing deviceaccording to claim 3, wherein the calculation unit is configured tocalculate, for each vicinity region, a local similarity degree betweenthe comparative frame and the reference frame, and calculate the totalsimilarity degree only using vicinity regions having a high localsimilarity degree.
 6. The image processing device according to claim 2,wherein the setting unit is configured to accept, from a user, settingof an area in a frame screen, set an image in the area of the specificframe included in the moving image or a frame that is obtained bycombining images in the area of the two or more specific frames includedin the moving image, as the reference frame, and set an image in thearea of the other specific frame included in the moving image as thecomparative frame.
 7. The image processing device according to claim 2,wherein the setting unit is capable of setting a plurality of referenceframes, and the calculation unit is configured to execute, if theplurality of reference frames are set, the calculation processing oneach of the reference frames, and the determination unit is configuredto determine, in the determination processing, that the comparativeframe and the reference frame or the frames combined into the referenceframe that have the highest total similarity degree belong to the samechannel.
 8. The image processing device according to claim 7, whereinthe setting unit is configured to sequentially select frames from themoving image, and set the first selected frame as the reference frameand a frame selected thereafter as the comparative frame, each time thesetting unit is configured to newly set a comparative frame, thecalculation unit is configured to execute the calculation processing andthe determination unit is configured to execute the determinationprocessing, and if it is determined, in the determination processing,that the comparative frame does not belong to the same channel as any ofthe reference frames or any of the frames combined into the referenceframes, the setting unit again is configured to set that comparativeframe as the reference frame.
 9. The image processing device accordingto claim 7, wherein the setting unit is configured to sequentiallyselect frames from the moving image, and set the first selected frame asthe reference frame and a frame selected thereafter as the comparativeframe, each time the setting unit newly is configured to set acomparative frame, the calculation unit is configured to execute thecalculation processing and the determination unit is configured toexecute the determination processing, and if it is determined, in thedetermination processing, that the comparative frame belongs to the samechannel as any of the reference frames or any of the frames combinedinto the reference frames, the setting unit is configured to combinethat comparative frame with the reference frame.
 10. The imageprocessing device according to claim 8, wherein, if the plurality ofreference frames are set, after the final frame has been selected fromthe moving image and the calculation processing and the determinationprocessing have ended, the calculation unit is configured to calculate asimilarity degree between the plurality of reference frames, and thedetermination unit is configured to combine channels that correspond toreference frames that have a high similarity degree between theplurality of reference frames into one channel.
 11. The image processingdevice according to claim 8, wherein, alter the final frame has beenselected from the moving image and the calculation processing and thedetermination processing have ended, the determination unit isconfigured to delete the channel to which only a predetermined number orless of frames belong.
 12. The image processing device according toclaim 1, wherein the similarity degree is a correlation coefficient. 13.A non-transitory computer readable medium storing an image processingprogram for sorting frames included in one moving image into differentchannels, the frames included in the moving image being a mixture offrames belonging to the different channels, the image processing programcausing a computer to execute a sorting step of calculating a similaritydegree of the plurality of frames included in the moving image byperforming image processing on the frames, and sorting the plurality offrames into the plurality of channels based on the similarity degree.14. The non-transitory computer readable medium according to claim 13,wherein the sorting step includes the steps of: setting a specific frameincluded in the moving image or a frame that is obtained by combiningtwo or more specific frames included in the moving image as a referenceframe; setting another specific frame included in the moving image as acomparative frame; dividing the comparative frame and the referenceframe into a plurality of sub-regions; calculating a local similaritydegree between the comparative frame and the reference frame for eachsub-region; calculating a total similarity degree, which is aframe-overall similarity degree, between the comparative frame and thereference frame only using sub-regions having a high local similaritydegree; and executing determination processing of determining, based onthe total similarity degree, whether or sot The comparative framebelongs to the same channel as that of the reference frame or the framescombined into the reference frame.
 15. The image processing programnon-transitory computer readable medium according to claim 13, whereinthe sorting step includes the steps of: setting a specific frameincluded in the moving image or a frame that is obtained by combiningtwo or more specific frames included in the moving image as a referenceframe; setting another specific frame included in the moving image as acomparative frame; detecting feature points from the comparative frame;setting a vicinity region for each feature point; calculating a totalsimilarity degree, which is a frame-overall similarity degree, betweenthe comparative frame and the reference frame only using the vicinityregions; and determining, based on the total similarity degree, whetheror not the comparative frame belongs to the same channel as that of thereference frame or the frames combined into the reference frame.
 16. Animage processing method for sorting frames included in one moving imageinto different channels, the frames included in the moving image being amixture of frames belonging to the different channels, the methodcomprising the step of calculating a similarity degree of the pluralityof frames included in the moving image by performing image processing onthe frames, and sorting the plurality of frames into the plurality ofchannels based on the similarity degree.
 17. The image processing deviceaccording to claim 4, wherein the calculation unit is configured tocalculate, for each vicinity region, a local similarity degree betweenthe comparative frame and the reference frame, and calculate the totalsimilarity degree only using vicinity regions having a high localsimilarity degree.
 18. The image processing device according to claim 3,wherein the setting unit is configured to accept, from a user, settingof an area in a frame screen, set an image in the area of the specificframe included in the moving image or a frame that is obtained bycombining images in the area of the two or more specific frames includedin the moving image, as the reference frame, and set an image in thearea of the other specific frame included in the moving image as thecomparative frame.
 19. The image processing device according to claim 4,wherein the setting unit is configured to accept, from a user, settingof an area in a frame screen, set an image in the area of the specificframe included in the moving image or a frame that is obtained bycombining images in the area of the two or more specific frames includedin the moving image, as the reference frame, and set an image in thearea of the other specific frame included in the moving image as thecomparative frame.
 20. The image processing device according to claim 5,wherein the setting unit is configured to accept, from a user, settingof an area in a frame screen, set an image in the area of the specificframe included in the moving image or a frame that is obtained bycombining images in the area of the two or more specific frames includedin the moving image, as the reference frame, and set an image in thearea of the other specific frame included in the moving image as thecomparative frame.